1
$\begingroup$

I have asked in another question how $\text{Var}{(\hat{y}_h)} = \sigma^2 \left(\frac{1}{n} + \frac{(x_h-\bar{x})^2}{S_{xx}}\right)$. Note that $\hat{y}_h$ = $b_0 + b_1X_h$ which is a regression line estimate at some given $X_h$.

This question concerns why the term $Cov(b_0,b_1)$ alone yields the RHS. Substituting $b_0 = Y - b_1X$ we get that $Cov(Y,b_1) - XCov(b_1,b_1)$ = $Cov(\frac{\sum{Y_i}}{n},\sum k_iY_i) - XVar{(b_1)}$. Here X and Y without subscript are arithmetic means.

We can then rearrange to obtain $\sum \frac{k_i Var(Y_i)}{n} - \frac{X\sigma^2}{S_{xx}}$ which quickly yields the desired result. My question is, why does this work? This single term does not seem like it should alone yield the RHS. Have I made an error in algebra?

$\endgroup$
  • $\begingroup$ Could anyone explain why this was downvoted, it seems like a reasonable question to me. $\endgroup$ – user26091 May 29 '13 at 14:33
  • 1
    $\begingroup$ I would start by searching our site: this formula has been quoted extensively and therefore likely has several different demonstrations. $\endgroup$ – whuber Nov 1 '18 at 19:17
  • $\begingroup$ Also related: stats.stackexchange.com/questions/115011/…. $\endgroup$ – StubbornAtom Nov 1 '18 at 19:20
2
$\begingroup$

$(1)\ E(\hat{Y_h}) = E(b_0 + b_1X_h) = \beta_0 +\beta_1X_h$

$(2)\ var(\hat{Y_h}) = var(b_0 + b_1X_h)$

An alternate (but equivalent) version of the regression model can be written as:

$Y_i = \beta_0X_0 + \beta_1X_1 + \epsilon_i$

This model associates an X variable with each coefficient $(where X_0 = 1)$

Al alternate modification is to use the deviation $X_i -\bar{X}$ rather than $X_i$

So the model can be written as:

$Y_i = \beta_0^* + \beta_1(X_i - \bar{X}) + \epsilon_i$

where $(3)\ \beta_0^* = \beta_0 + \beta_1\bar{X}$

These models can be used interchangably.

We know from the normal equations:

$\Sigma Y_i = nb_0 + b_1\Sigma X_i$

solving for $b_0$

$(4)\ b_0 = \bar{Y} - b_1\bar{X}$

So substituting from (3) and (4):

$b_0^* = b_0 + b_1\bar{X} = (\bar{Y} - b_1\bar{X}) + b_1\bar{X} = \bar{Y}$

$(5)\ var(\hat{Y_h}) = var(b_0 + b_1X_h) = var(\bar{Y} + b_1(X_h - \bar{X}))$

using:

$var(\bar{Y}) = \frac{\sigma^2}{n}$

$var(aX) = a^2var(X)$

and

$var(X + Y) = Var(X) + Var(Y) + 2Cov(X,Y)$

So:

= $var(\bar{Y}) +(X_h - \bar{X})^2var(b_1) + 2(X_h-\bar{X})cov(\bar{Y},b_1)$

we use the fact that $Cov(\bar{Y},b_1) = 0$ due to the i.i.d assumption on $\epsilon_i$

$= \frac{\sigma^2}{n} + (X_h-\bar{X})^2\frac{\sigma^2}{\Sigma(X_i-\bar{X})^2}$

$= \sigma^2[\frac{1}{n} + \frac{(X_h - \bar{X})^2}{\Sigma(X_i - \bar{X})^2}]$

$\endgroup$
  • $\begingroup$ All of this can be found in Chapter 1 and 2 of Applied Linear Regression Models by Kutner, Nachtsheim and Neter. That's where I was sifting through to find stuff. $\endgroup$ – Eric Peterson May 29 '13 at 12:49
  • $\begingroup$ including several pages on the normal equations which shows how $b_1 = \frac{\Sigma(X_i - \bar{X})(Y_i - \bar{Y})}{\Sigma(X_i - \bar{X})^2}$ to get you the last two steps of my proof. $\endgroup$ – Eric Peterson May 29 '13 at 13:07
  • $\begingroup$ Thank you Clark, very clear. I will see if I can find that book. Enjoy your trip to Wally World this summer. $\endgroup$ – user26091 May 29 '13 at 15:34
1
$\begingroup$

Provided that

$$ \text{Var}(\hat{y}_{x_0}) = \displaystyle\frac{\sigma^2\sum x_i^2}{n\sum(x_i - \bar{x})^2} + \displaystyle\frac{\sigma^2x_0^2}{\sum(x_i - \bar{x})^2} - \displaystyle\frac{2x_0\sigma^2\bar{x}}{\sum(x_i - \bar{x})^2} $$

The term, $\displaystyle\frac{\sigma^2\sum x_i^2}{n\sum(x_i - \bar{x})^2}$ is the most troublesome and can be broken down using basic arithmetic and knowing that playing with $\sum x_i^2$ is dangerous. So,

$$ \displaystyle\frac{\sigma^2(\sum x_i^2 + 2n\bar{x}^2 - 2n\bar{x}^2)}{n\sum(x_i - \bar{x})^2} \rightarrow \displaystyle\frac{\sigma^2(n\bar{x}^2 + \sum x_i^2 + \bar{x}^2\sum(1) - 2n\bar{x}\{\sum(x_i)/n\})}{n\sum(x_i - \bar{x})^2} $$ Where $n = \sum(1)$ and $\bar{x} = \sum(x_i)/n$ which is basic. $$ \displaystyle\frac{\sigma^2(n\bar{x}^2 + \sum(x_i^2 + \bar{x}^2 - 2\bar{x}x_i))}{n\sum(x_i - \bar{x})^2} \rightarrow \displaystyle\frac{\sigma^2(n\bar{x}^2 + \sum(x_i - \bar{x})^2)}{n\sum(x_i - \bar{x})^2} $$ Here $\sum(x_i - \bar{x})^2 = S_{xx}$ so, our final term is $$ \displaystyle\frac{\sigma^2(n\bar{x}^2 + S_{xx})}{nS_{xx}} \rightarrow \displaystyle\frac{\sigma^2\bar{x}^2}{S_{xx}} + \displaystyle\frac{\sigma^2}{n} $$ So, our original equation becomes $$ \text{Var}(\hat{y}_{x_0}) = \displaystyle\frac{\sigma^2\bar{x}^2}{S_{xx}} + \displaystyle\frac{\sigma^2}{n} + \displaystyle\frac{\sigma^2x_0^2}{S_{xx}} - \displaystyle\frac{2x_0\sigma^2\bar{x}}{S_{xx}} $$ $$ \text{Var}(\hat{y}_{x_0}) = \sigma^2\left[\displaystyle\frac{1}{n} + \displaystyle\frac{\bar{x}^2 + x_0^2-2x_0\bar{x}}{S_{xx}}\right] \rightarrow \text{Var}(\hat{y}_{x_0}) = \sigma^2\left[\displaystyle\frac{1}{n} + \displaystyle\frac{(x_0 - \bar{x})^2}{S_{xx}}\right] $$

$\endgroup$
  • $\begingroup$ Hello and thanks for your input ! I think that this $$\text{Var}(\hat{y}_{x_0}) = \displaystyle\frac{\sigma^2\sum x_i^2}{n\sum(x_i - \bar{x})^2} + \displaystyle\frac{\sigma^2x_0^2}{\sum(x_i - \bar{x})^2} - \displaystyle\frac{2x_0\sigma^2\bar{x}}{\sum(x_i - \bar{x})^2}$$ needs to be proven as well though. $\endgroup$ – Rebellos Nov 2 '18 at 8:24
  • $\begingroup$ My bad, I'll see to it ASAP. $\endgroup$ – Syed Ali Nov 2 '18 at 10:41
  • $\begingroup$ Did you check on how to prove the big expression? $\endgroup$ – Rebellos Nov 3 '18 at 15:18
  • $\begingroup$ No luck with that one $\endgroup$ – Syed Ali Nov 4 '18 at 12:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.