I know the following results:

$\frac{RSS}{\sigma^{2}} \sim \chi^{2}_{n-r}$ where $r = rank(X)$ and $k$ is the degrees of freedom. Then, for any $c \in \mathbb{R^{p}} $, and when X has full rank, i.e. $rank(X) = p$, $\frac{\mathbf{c}^{T}\hat{\beta}-\mathbf{c}^{T}\beta}{\sqrt{\mathbf{c^{T}(X^{T}X)^{-1}\mathbf{c}\frac{RSS}{n-p}}}} \sim t_{n-p}$, following from $\frac{\mathbf{c^{T}}\hat{\beta}-\mathbf{c^{T}}\beta}{\mathbf{c^{T}}(X^{T}X)^{-1}\mathbf{c}\sigma^{2}} \sim N(0,1)$.

Suppose that we fit a linear model with design matrix X to obtain estimates $\boldsymbol{\hat{\beta}}$ and $\hat{\sigma}^{2}$. For given covariates, $x_{*}$, the predicted response is $\hat{y}_{*} = x_{*}^{T}\boldsymbol{\hat{\beta}}$.

We also know that $Var(x_{*}^{T}\hat{\beta}) = x_{*}^{T}(X^{T}X)^{-1}x_{*}\sigma^{2}$.

How, from the above, can I get the following result? A $100(1-\alpha)$ confidence interval for a single future response is $\hat{y_{*}} \pm t_{n-p}^{(\frac{\alpha}{2})}\hat{\sigma}\sqrt{1+x_{*}^{T}(X^{T}X)^{-1}x_{*}}$? I've been stuck for ages.

  • $\begingroup$ To be precise, a confidence interval is an interval for the true value of a fixed population quantity, such as a parameter. It sounds like what you want is a prediction interval. $\endgroup$ – Kodiologist Feb 6 '17 at 19:47
  • $\begingroup$ @Kodiologist Thank you - I changed the mistake. $\endgroup$ – python_learner Feb 6 '17 at 19:50

If you wonder where did the extra $1$ come from. We have

$$E(\hat{Y}_* - Y_*)^2 = Var(\hat{Y}_*) + E(E(\hat{Y}_*) - Y_*)^2 = Var(\hat{Y}_*) + E(Y_* - x_*\beta)^2 = Var(\hat{Y}_*) + \sigma^2$$

Let's denote $Z = Y_* - \hat{Y}_*$. From Fisher's Theorem we know that $\hat{Y}_*$ is normally distributed, then so is $Z$. Now

$$\frac{Z - E(Z)}{\sqrt{Var(Z)}} = \frac{Y_* - \hat{Y}_* - 0}{\sqrt{E(\hat{Y}_* - Y_* - 0)^2}} = \frac{Y_* - \hat{Y}_*}{\sqrt{\sigma^2D_*}} \sim N(0,1)$$

Where $D_* = 1 + x_*^T(X^TX)^{-1}x_*$. Substituing estimator of $\sigma^2$, $S^2 = \frac{RSS}{n-p}$

$$\frac{Y_* - \hat{Y}_*}{\sqrt{D_*\frac{RSS}{n-p}}} \sim t(n-p)$$

and finally

$$P\Big(t_{\alpha/2} \le \frac{Y_* - \hat{Y}_*}{\sqrt{D_*\frac{RSS}{n-p}}} \le t_{1-\alpha/2}\Big) = 1 - \alpha$$

$$P\Big(\hat{Y}_*-t_{1-\alpha/2}\sqrt{D_*\frac{RSS}{n-p}} \le Y_* \le \hat{Y}_*+t_{1-\alpha/2}\sqrt{D_*\frac{RSS}{n-p}}\Big) = 1 - \alpha$$

| cite | improve this answer | |
  • $\begingroup$ Do you mind explaining a little further? I was kind of looking for a full explanation (derivation) from the above results. $\endgroup$ – python_learner Feb 6 '17 at 19:36
  • $\begingroup$ @python_learner I hope it's clear now $\endgroup$ – Łukasz Grad Feb 6 '17 at 21:08

Prediction intervals are wider because there is that extra uncertainty due to the new random observation. See Hahn and Meeker's Statistical Intervals text published by Wiley for clear derivations and discussions of the various types of statistical intervals including Bayesian credible regions.

| cite | improve this answer | |
  • $\begingroup$ I was looking for the derivation mostly - could you provide it or at least point to a place in the text I can find it at? $\endgroup$ – python_learner Feb 6 '17 at 19:38
  • 1
    $\begingroup$ While researching this I am sorry to say that my friend Gerry Hahn passed away. $\endgroup$ – Michael R. Chernick Feb 6 '17 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.