Suppose a linear model has been fitted with design matrix X and there are estimates $\boldsymbol{\hat{\beta}}$ and $\hat{\sigma}^{2}$. For given covariates, $x_{*}$, the predicted response is $\hat{y}_{*} = x_{*}^{T}\boldsymbol{\hat{\beta}}$.

Here is the bit I don't get: why is $Var(x_{*}^{T}\boldsymbol{\hat{\beta}}) = x_{*}^{T}(X^{T}X)^{-1}x_{*}\sigma^{2}$? How do you show that?

  • $\begingroup$ First you have to show that $Var(\hat{\beta}) = (X^TX)^{-1}\sigma^2$ simply from the equation for $\hat{\beta} = (X^TX)^{-1}X^TY$. Keep in mind that $E[\hat{\beta}] = (X^TX)^{-1}X^TE[Y] = (X^TX)^{-1}X^TX\beta = \beta$ $\endgroup$ – Łukasz Grad Jan 30 '17 at 20:58
  • $\begingroup$ Is this self-study? If so, please add the tag. $\endgroup$ – Richard Hardy Jan 30 '17 at 21:07
  • $\begingroup$ @RichardHardy It's not a question posed in a test or a textbook or anything; it was simply a statement from my lecture notes that my lecturer glossed over - does that count as self-study too? $\endgroup$ – python_learner Jan 30 '17 at 21:17
  • $\begingroup$ @ŁukaszGrad Yep, I got that part. So in showing that, would I use the formula for the variance of the product of two random variables? $\endgroup$ – python_learner Jan 31 '17 at 13:22
  • $\begingroup$ @python_learner No special formula, because $x_*$ is known, it's not a r.v. Just calculate again $Var(x_*^T\hat{\beta}) = E[x_*^T\hat{\beta} - E[x_*^T\hat{\beta}]][x_*^T\hat{\beta} - E[x_*^T\hat{\beta}]]^T$ $\endgroup$ – Łukasz Grad Jan 31 '17 at 13:54

Under the assumptions:

  • Expected value of the errors equal to zero.
  • Constant variance of the errors.
  • Independence of the errors.
  • Normality of the response.

You have



$\hat{Y}=X(X^tX)^{-1}X^tY=HY$, where $H^2=H$ and $H=H^t$.


Than you have:

$\Sigma_{\hat{Y}}=E[(\hat{Y}-X\beta)(\hat{Y}-X\beta)^t]=E[(Y-\epsilon+H\epsilon-Y+\epsilon)(Y-\epsilon+H\epsilon-Y+\epsilon)^t]= E[H\epsilon\epsilon^tH]=HE[\epsilon\epsilon^t]H=H\sigma^2I=X(X^tX)^{-1}X^t\sigma^2I$


If you take the elements of the diagonal of $\Sigma_{\hat{Y}}$, than you find the variance of each element of the vector $\hat{Y}$.

  • $\begingroup$ What is the $\Sigma_{\hat{Y}} $ meant to represent? $\endgroup$ – python_learner Jan 30 '17 at 21:56
  • $\begingroup$ $\Sigma = E[( X − E [ X ] ) ( X − E [ X ] ) ^T ]$ This is the Covariance Matrix and it can be seen as a generalization of the scalar-valued variance to higher dimensions. Recall that for a scalar-valued random variable X $\sigma^ 2 = v a r ( X ) = E [ ( X − E ( X ) ) ^2 ] $ Indeed, the entries on the diagonal of the covariance matrix are the variances of each element of the vector $X$. $\endgroup$ – momomi Jan 30 '17 at 22:13
  • $\begingroup$ I made some mistakes in the previous answer. Just improved it. $\endgroup$ – momomi Jan 31 '17 at 9:58
  • $\begingroup$ @python_learner Is my answer clear? I just used the matrix notation but the demonstration is basically the same. $\endgroup$ – momomi Feb 1 '17 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.