# Variance in linear regression model

Suppose a linear model has been fitted with design matrix X and there are estimates $\boldsymbol{\hat{\beta}}$ and $\hat{\sigma}^{2}$. For given covariates, $x_{*}$, the predicted response is $\hat{y}_{*} = x_{*}^{T}\boldsymbol{\hat{\beta}}$.

Here is the bit I don't get: why is $Var(x_{*}^{T}\boldsymbol{\hat{\beta}}) = x_{*}^{T}(X^{T}X)^{-1}x_{*}\sigma^{2}$? How do you show that?

• First you have to show that $Var(\hat{\beta}) = (X^TX)^{-1}\sigma^2$ simply from the equation for $\hat{\beta} = (X^TX)^{-1}X^TY$. Keep in mind that $E[\hat{\beta}] = (X^TX)^{-1}X^TE[Y] = (X^TX)^{-1}X^TX\beta = \beta$ – Łukasz Grad Jan 30 '17 at 20:58
• Is this self-study? If so, please add the tag. – Richard Hardy Jan 30 '17 at 21:07
• @RichardHardy It's not a question posed in a test or a textbook or anything; it was simply a statement from my lecture notes that my lecturer glossed over - does that count as self-study too? – python_learner Jan 30 '17 at 21:17
• @ŁukaszGrad Yep, I got that part. So in showing that, would I use the formula for the variance of the product of two random variables? – python_learner Jan 31 '17 at 13:22
• @python_learner No special formula, because $x_*$ is known, it's not a r.v. Just calculate again $Var(x_*^T\hat{\beta}) = E[x_*^T\hat{\beta} - E[x_*^T\hat{\beta}]][x_*^T\hat{\beta} - E[x_*^T\hat{\beta}]]^T$ – Łukasz Grad Jan 31 '17 at 13:54

Under the assumptions:

• Expected value of the errors equal to zero.
• Constant variance of the errors.
• Independence of the errors.
• Normality of the response.

You have

$\Sigma_{\epsilon}=E[(\epsilon-0)(\epsilon-0)^t]=E[\epsilon\epsilon^t]=\sigma^2I$

$E[\hat{Y}]=X\beta$

$\hat{Y}=X(X^tX)^{-1}X^tY=HY$, where $H^2=H$ and $H=H^t$.

$\hat{Y}=X(X^tX)^{-1}X^tY=X(X^tX)^{-1}X^t(X\beta+\epsilon)=(Y-\epsilon)+H\epsilon$

Than you have:

$\Sigma_{\hat{Y}}=E[(\hat{Y}-X\beta)(\hat{Y}-X\beta)^t]=E[(Y-\epsilon+H\epsilon-Y+\epsilon)(Y-\epsilon+H\epsilon-Y+\epsilon)^t]= E[H\epsilon\epsilon^tH]=HE[\epsilon\epsilon^t]H=H\sigma^2I=X(X^tX)^{-1}X^t\sigma^2I$

.

If you take the elements of the diagonal of $\Sigma_{\hat{Y}}$, than you find the variance of each element of the vector $\hat{Y}$.

• What is the $\Sigma_{\hat{Y}}$ meant to represent? – python_learner Jan 30 '17 at 21:56
• $\Sigma = E[( X − E [ X ] ) ( X − E [ X ] ) ^T ]$ This is the Covariance Matrix and it can be seen as a generalization of the scalar-valued variance to higher dimensions. Recall that for a scalar-valued random variable X $\sigma^ 2 = v a r ( X ) = E [ ( X − E ( X ) ) ^2 ]$ Indeed, the entries on the diagonal of the covariance matrix are the variances of each element of the vector $X$. – momomi Jan 30 '17 at 22:13
• I made some mistakes in the previous answer. Just improved it. – momomi Jan 31 '17 at 9:58
• @python_learner Is my answer clear? I just used the matrix notation but the demonstration is basically the same. – momomi Feb 1 '17 at 11:04