3
$\begingroup$

I have asked in another question how $\text{Var}{(\hat{y}_h)} = \sigma^2 \left(\frac{1}{n} + \frac{(x_h-\bar{x})^2}{S_{xx}}\right)$. Note that $\hat{y}_h$ = $b_0 + b_1X_h$ which is a regression line estimate at some given $X_h$.

This question concerns why the term $Cov(b_0,b_1)$ alone yields the RHS. Substituting $b_0 = Y - b_1X$ we get that $Cov(Y,b_1) - XCov(b_1,b_1)$ = $Cov(\frac{\sum{Y_i}}{n},\sum k_iY_i) - XVar{(b_1)}$. Here X and Y without subscript are arithmetic means.

We can then rearrange to obtain $\sum \frac{k_i Var(Y_i)}{n} - \frac{X\sigma^2}{S_{xx}}$ which quickly yields the desired result. My question is, why does this work? This single term does not seem like it should alone yield the RHS. Have I made an error in algebra?

$\endgroup$
3
  • 1
    $\begingroup$ Could anyone explain why this was downvoted, it seems like a reasonable question to me. $\endgroup$
    – user26091
    May 29, 2013 at 14:33
  • 1
    $\begingroup$ I would start by searching our site: this formula has been quoted extensively and therefore likely has several different demonstrations. $\endgroup$
    – whuber
    Nov 1, 2018 at 19:17
  • $\begingroup$ Also related: stats.stackexchange.com/questions/115011/…. $\endgroup$ Nov 1, 2018 at 19:20

2 Answers 2

3
$\begingroup$

$(1)\ E(\hat{Y_h}) = E(b_0 + b_1X_h) = \beta_0 +\beta_1X_h$

$(2)\ var(\hat{Y_h}) = var(b_0 + b_1X_h)$

An alternate (but equivalent) version of the regression model can be written as:

$Y_i = \beta_0X_0 + \beta_1X_1 + \epsilon_i$

This model associates an X variable with each coefficient $(where X_0 = 1)$

Al alternate modification is to use the deviation $X_i -\bar{X}$ rather than $X_i$

So the model can be written as:

$Y_i = \beta_0^* + \beta_1(X_i - \bar{X}) + \epsilon_i$

where $(3)\ \beta_0^* = \beta_0 + \beta_1\bar{X}$

These models can be used interchangably.

We know from the normal equations:

$\Sigma Y_i = nb_0 + b_1\Sigma X_i$

solving for $b_0$

$(4)\ b_0 = \bar{Y} - b_1\bar{X}$

So substituting from (3) and (4):

$b_0^* = b_0 + b_1\bar{X} = (\bar{Y} - b_1\bar{X}) + b_1\bar{X} = \bar{Y}$

$(5)\ var(\hat{Y_h}) = var(b_0 + b_1X_h) = var(\bar{Y} + b_1(X_h - \bar{X}))$

using:

$var(\bar{Y}) = \frac{\sigma^2}{n}$

$var(aX) = a^2var(X)$

and

$var(X + Y) = Var(X) + Var(Y) + 2Cov(X,Y)$

So:

= $var(\bar{Y}) +(X_h - \bar{X})^2var(b_1) + 2(X_h-\bar{X})cov(\bar{Y},b_1)$

we use the fact that $Cov(\bar{Y},b_1) = 0$ due to the i.i.d assumption on $\epsilon_i$

$= \frac{\sigma^2}{n} + (X_h-\bar{X})^2\frac{\sigma^2}{\Sigma(X_i-\bar{X})^2}$

$= \sigma^2[\frac{1}{n} + \frac{(X_h - \bar{X})^2}{\Sigma(X_i - \bar{X})^2}]$

$\endgroup$
3
  • $\begingroup$ All of this can be found in Chapter 1 and 2 of Applied Linear Regression Models by Kutner, Nachtsheim and Neter. That's where I was sifting through to find stuff. $\endgroup$ May 29, 2013 at 12:49
  • $\begingroup$ including several pages on the normal equations which shows how $b_1 = \frac{\Sigma(X_i - \bar{X})(Y_i - \bar{Y})}{\Sigma(X_i - \bar{X})^2}$ to get you the last two steps of my proof. $\endgroup$ May 29, 2013 at 13:07
  • $\begingroup$ Thank you Clark, very clear. I will see if I can find that book. Enjoy your trip to Wally World this summer. $\endgroup$
    – user26091
    May 29, 2013 at 15:34
1
$\begingroup$

Provided that

$$ \text{Var}(\hat{y}_{x_0}) = \displaystyle\frac{\sigma^2\sum x_i^2}{n\sum(x_i - \bar{x})^2} + \displaystyle\frac{\sigma^2x_0^2}{\sum(x_i - \bar{x})^2} - \displaystyle\frac{2x_0\sigma^2\bar{x}}{\sum(x_i - \bar{x})^2} $$

The term, $\displaystyle\frac{\sigma^2\sum x_i^2}{n\sum(x_i - \bar{x})^2}$ is the most troublesome and can be broken down using basic arithmetic and knowing that playing with $\sum x_i^2$ is dangerous. So,

$$ \displaystyle\frac{\sigma^2(\sum x_i^2 + 2n\bar{x}^2 - 2n\bar{x}^2)}{n\sum(x_i - \bar{x})^2} \rightarrow \displaystyle\frac{\sigma^2(n\bar{x}^2 + \sum x_i^2 + \bar{x}^2\sum(1) - 2n\bar{x}\{\sum(x_i)/n\})}{n\sum(x_i - \bar{x})^2} $$ Where $n = \sum(1)$ and $\bar{x} = \sum(x_i)/n$ which is basic. $$ \displaystyle\frac{\sigma^2(n\bar{x}^2 + \sum(x_i^2 + \bar{x}^2 - 2\bar{x}x_i))}{n\sum(x_i - \bar{x})^2} \rightarrow \displaystyle\frac{\sigma^2(n\bar{x}^2 + \sum(x_i - \bar{x})^2)}{n\sum(x_i - \bar{x})^2} $$ Here $\sum(x_i - \bar{x})^2 = S_{xx}$ so, our final term is $$ \displaystyle\frac{\sigma^2(n\bar{x}^2 + S_{xx})}{nS_{xx}} \rightarrow \displaystyle\frac{\sigma^2\bar{x}^2}{S_{xx}} + \displaystyle\frac{\sigma^2}{n} $$ So, our original equation becomes $$ \text{Var}(\hat{y}_{x_0}) = \displaystyle\frac{\sigma^2\bar{x}^2}{S_{xx}} + \displaystyle\frac{\sigma^2}{n} + \displaystyle\frac{\sigma^2x_0^2}{S_{xx}} - \displaystyle\frac{2x_0\sigma^2\bar{x}}{S_{xx}} $$ $$ \text{Var}(\hat{y}_{x_0}) = \sigma^2\left[\displaystyle\frac{1}{n} + \displaystyle\frac{\bar{x}^2 + x_0^2-2x_0\bar{x}}{S_{xx}}\right] \rightarrow \text{Var}(\hat{y}_{x_0}) = \sigma^2\left[\displaystyle\frac{1}{n} + \displaystyle\frac{(x_0 - \bar{x})^2}{S_{xx}}\right] $$

$\endgroup$
4
  • $\begingroup$ Hello and thanks for your input ! I think that this $$\text{Var}(\hat{y}_{x_0}) = \displaystyle\frac{\sigma^2\sum x_i^2}{n\sum(x_i - \bar{x})^2} + \displaystyle\frac{\sigma^2x_0^2}{\sum(x_i - \bar{x})^2} - \displaystyle\frac{2x_0\sigma^2\bar{x}}{\sum(x_i - \bar{x})^2}$$ needs to be proven as well though. $\endgroup$
    – Rebellos
    Nov 2, 2018 at 8:24
  • $\begingroup$ My bad, I'll see to it ASAP. $\endgroup$ Nov 2, 2018 at 10:41
  • $\begingroup$ Did you check on how to prove the big expression? $\endgroup$
    – Rebellos
    Nov 3, 2018 at 15:18
  • $\begingroup$ No luck with that one $\endgroup$ Nov 4, 2018 at 12:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.