1
$\begingroup$

I know the following results:

$\frac{RSS}{\sigma^{2}} \sim \chi^{2}_{n-r}$ where $r = rank(X)$ and $k$ is the degrees of freedom. Then, for any $c \in \mathbb{R^{p}} $, and when X has full rank, i.e. $rank(X) = p$, $\frac{\mathbf{c}^{T}\hat{\beta}-\mathbf{c}^{T}\beta}{\sqrt{\mathbf{c^{T}(X^{T}X)^{-1}\mathbf{c}\frac{RSS}{n-p}}}} \sim t_{n-p}$, following from $\frac{\mathbf{c^{T}}\hat{\beta}-\mathbf{c^{T}}\beta}{\mathbf{c^{T}}(X^{T}X)^{-1}\mathbf{c}\sigma^{2}} \sim N(0,1)$.

Suppose that we fit a linear model with design matrix X to obtain estimates $\boldsymbol{\hat{\beta}}$ and $\hat{\sigma}^{2}$. For given covariates, $x_{*}$, the predicted response is $\hat{y}_{*} = x_{*}^{T}\boldsymbol{\hat{\beta}}$.

We also know that $Var(x_{*}^{T}\hat{\beta}) = x_{*}^{T}(X^{T}X)^{-1}x_{*}\sigma^{2}$.

How, from the above, can I get the following result? A $100(1-\alpha)$ confidence interval for a single future response is $\hat{y_{*}} \pm t_{n-p}^{(\frac{\alpha}{2})}\hat{\sigma}\sqrt{1+x_{*}^{T}(X^{T}X)^{-1}x_{*}}$? I've been stuck for ages.

$\endgroup$
2
  • $\begingroup$ To be precise, a confidence interval is an interval for the true value of a fixed population quantity, such as a parameter. It sounds like what you want is a prediction interval. $\endgroup$ Commented Feb 6, 2017 at 19:47
  • $\begingroup$ @Kodiologist Thank you - I changed the mistake. $\endgroup$ Commented Feb 6, 2017 at 19:50

2 Answers 2

4
$\begingroup$

If you wonder where did the extra $1$ come from. We have

$$E(\hat{Y}_* - Y_*)^2 = Var(\hat{Y}_*) + E(E(\hat{Y}_*) - Y_*)^2 = Var(\hat{Y}_*) + E(Y_* - x_*\beta)^2 = Var(\hat{Y}_*) + \sigma^2$$

Let's denote $Z = Y_* - \hat{Y}_*$. From Fisher's Theorem we know that $\hat{Y}_*$ is normally distributed, then so is $Z$. Now

$$\frac{Z - E(Z)}{\sqrt{Var(Z)}} = \frac{Y_* - \hat{Y}_* - 0}{\sqrt{E(\hat{Y}_* - Y_* - 0)^2}} = \frac{Y_* - \hat{Y}_*}{\sqrt{\sigma^2D_*}} \sim N(0,1)$$

Where $D_* = 1 + x_*^T(X^TX)^{-1}x_*$. Substituing estimator of $\sigma^2$, $S^2 = \frac{RSS}{n-p}$

$$\frac{Y_* - \hat{Y}_*}{\sqrt{D_*\frac{RSS}{n-p}}} \sim t(n-p)$$

and finally

$$P\Big(t_{\alpha/2} \le \frac{Y_* - \hat{Y}_*}{\sqrt{D_*\frac{RSS}{n-p}}} \le t_{1-\alpha/2}\Big) = 1 - \alpha$$

$$P\Big(\hat{Y}_*-t_{1-\alpha/2}\sqrt{D_*\frac{RSS}{n-p}} \le Y_* \le \hat{Y}_*+t_{1-\alpha/2}\sqrt{D_*\frac{RSS}{n-p}}\Big) = 1 - \alpha$$

$\endgroup$
2
  • $\begingroup$ Do you mind explaining a little further? I was kind of looking for a full explanation (derivation) from the above results. $\endgroup$ Commented Feb 6, 2017 at 19:36
  • $\begingroup$ @python_learner I hope it's clear now $\endgroup$ Commented Feb 6, 2017 at 21:08
1
$\begingroup$

Prediction intervals are wider because there is that extra uncertainty due to the new random observation. See Hahn and Meeker's Statistical Intervals text published by Wiley for clear derivations and discussions of the various types of statistical intervals including Bayesian credible regions.

$\endgroup$
2
  • $\begingroup$ I was looking for the derivation mostly - could you provide it or at least point to a place in the text I can find it at? $\endgroup$ Commented Feb 6, 2017 at 19:38
  • 1
    $\begingroup$ While researching this I am sorry to say that my friend Gerry Hahn passed away. $\endgroup$ Commented Feb 6, 2017 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.