25
$\begingroup$

I know that $$\hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x}$$ and this is how far I got when I calculated the variance:

\begin{align*} Var(\hat{\beta_0}) &= Var(\bar{y} - \hat{\beta_1}\bar{x}) \\ &= Var((-\bar{x})\hat{\beta_1}+\bar{y}) \\ &= Var((-\bar{x})\hat{\beta_1})+Var(\bar{y}) \\ &= (-\bar{x})^2 Var(\hat{\beta_1}) + 0 \\ &= (\bar{x})^2 Var(\hat{\beta_1}) + 0 \\ &= \frac{\sigma^2 (\bar{x})^2}{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2} \end{align*}

but that's far as I got. The final formula I'm trying to calculate is

\begin{align*} Var(\hat{\beta_0}) &= \frac{\sigma^2 n^{-1}\displaystyle\sum\limits_{i=1}^n x_i^2}{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2} \end{align*}

I'm not sure how to get $$(\bar{x})^2 = \frac{1}{n}\displaystyle\sum\limits_{i=1}^n x_i^2$$ assuming my math is correct up to there.

Is this the right path?

\begin{align} (\bar{x})^2 &= \left(\frac{1}{n}\displaystyle\sum\limits_{i=1}^n x_i\right)^2 \\ &= \frac{1}{n^2} \left(\displaystyle\sum\limits_{i=1}^n x_i\right)^2 \end{align}

I'm sure it's simple, so the answer can wait for a bit if someone has a hint to push me in the right direction.

$\endgroup$
6
  • 2
    $\begingroup$ This is not the right path. The 4th equation doesn't hold. For example, with $x_1=−1$, $x_2=0$, and $x_3=1$, the left term is zero, whilst the right term is $2/3$. The problem comes from the step where you split the variance (3rd line of second equation). See why? $\endgroup$
    – QuantIbex
    Commented Jul 12, 2013 at 23:55
  • $\begingroup$ Hint towards Quantlbex point: variance is not a linear function. It violates both additivity and scalar multiplication. $\endgroup$
    – David Marx
    Commented Jul 13, 2013 at 1:53
  • $\begingroup$ @DavidMarx That step should be $$=Var((-\bar{x})\hat{\beta_1}+\bar{y})=(\bar{x})^2Var(\hat{\beta_1})+\bar{y}$$, I think, and then once I substitute in for $\hat{\beta_1}$ and $\bar{y}$ (not sure what to do for this but I'll think about it more), that should put me on the right path I hope. $\endgroup$
    – M T
    Commented Jul 13, 2013 at 3:53
  • $\begingroup$ This is not correct. Think about the condition required for the variance of a sum to be equal to the sum of the variances. $\endgroup$
    – QuantIbex
    Commented Jul 13, 2013 at 10:29
  • 2
    $\begingroup$ No, $\bar{y}$ is random since $y_i = \beta_0 + \beta_1 x_i + \epsilon$, where $\epsilon$ denotes the (random) noise. But OK, my previous comment was maybe misleading. Also, ${\rm Var}(aX + b)= a^2{\rm Var}(X)$, if $a$ and $b$ denote constants. $\endgroup$
    – QuantIbex
    Commented Jul 13, 2013 at 19:38

2 Answers 2

30
$\begingroup$

This is a self-study question, so I provide hints that will hopefully help to find the solution, and I'll edit the answer based on your feedbacks/progress.

The parameter estimates that minimize the sum of squares are \begin{align} \hat{\beta}_0 &= \bar{y} - \hat{\beta}_1 \bar{x} , \\ \hat{\beta}_1 &= \frac{ \sum_{i = 1}^n(x_i - \bar{x})y_i }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } . \end{align} To get the variance of $\hat{\beta}_0$, start from its expression and substitute the expression of $\hat{\beta}_1$, and do the algebra $$ {\rm Var}(\hat{\beta}_0) = {\rm Var} (\bar{Y} - \hat{\beta}_1 \bar{x}) = \ldots $$

Edit:
We have \begin{align} {\rm Var}(\hat{\beta}_0) &= {\rm Var} (\bar{Y} - \hat{\beta}_1 \bar{x}) \\ &= {\rm Var} (\bar{Y}) + (\bar{x})^2 {\rm Var} (\hat{\beta}_1) - 2 \bar{x} {\rm Cov} (\bar{Y}, \hat{\beta}_1). \end{align} The two variance terms are $$ {\rm Var} (\bar{Y}) = {\rm Var} \left(\frac{1}{n} \sum_{i = 1}^n Y_i \right) = \frac{1}{n^2} \sum_{i = 1}^n {\rm Var} (Y_i) = \frac{\sigma^2}{n}, $$ and \begin{align} {\rm Var} (\hat{\beta}_1) &= \frac{ 1 }{ \left[\sum_{i = 1}^n(x_i - \bar{x})^2 \right]^2 } \sum_{i = 1}^n(x_i - \bar{x})^2 {\rm Var} (Y_i) \\ &= \frac{ \sigma^2 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } , \end{align} and the covariance term is \begin{align} {\rm Cov} (\bar{Y}, \hat{\beta}_1) &= {\rm Cov} \left\{ \frac{1}{n} \sum_{i = 1}^n Y_i, \frac{ \sum_{j = 1}^n(x_j - \bar{x})Y_j }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } \right \} \\ &= \frac{1}{n} \frac{ 1 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } {\rm Cov} \left\{ \sum_{i = 1}^n Y_i, \sum_{j = 1}^n(x_j - \bar{x})Y_j \right\} \\ &= \frac{ 1 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \sum_{i = 1}^n (x_j - \bar{x}) \sum_{j = 1}^n {\rm Cov}(Y_i, Y_j) \\ &= \frac{ 1 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \sum_{i = 1}^n (x_j - \bar{x}) \sigma^2 \\ &= 0 \end{align} since $\sum_{i = 1}^n (x_j - \bar{x})=0$.
And since $$\sum_{i = 1}^n(x_i - \bar{x})^2 = \sum_{i = 1}^n x_i^2 - 2 \bar{x} \sum_{i = 1}^n x_i + \sum_{i = 1}^n \bar{x}^2 = \sum_{i = 1}^n x_i^2 - n \bar{x}^2, $$ we have \begin{align} {\rm Var}(\hat{\beta}_0) &= \frac{\sigma^2}{n} + \frac{ \sigma^2 \bar{x}^2}{ \sum_{i = 1}^n(x_i - \bar{x})^2 } \\ &= \frac{\sigma^2 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \left\{ \sum_{i = 1}^n(x_i - \bar{x})^2 + n \bar{x}^2 \right\} \\ &= \frac{\sigma^2 \sum_{i = 1}^n x_i^2}{ n \sum_{i = 1}^n(x_i - \bar{x})^2 }. \end{align}

Edit 2

Why do we have ${\rm var} ( \sum_{i = 1}^n Y_i) = \sum_{i = 1}^n {\rm Var} (Y_i) $?

The assumed model is $ Y_i = \beta_0 + \beta_1 X_i + \epsilon_i$, where the $\epsilon_i$ are independant and identically distributed random variables with ${\rm E}(\epsilon_i) = 0$ and ${\rm var}(\epsilon_i) = \sigma^2$.

Once we have a sample, the $X_i$ are known, the only random terms are the $\epsilon_i$. Recalling that for a random variable $Z$ and a constant $a$, we have ${\rm var}(a+Z) = {\rm var}(Z)$. Thus, \begin{align} {\rm var} \left( \sum_{i = 1}^n Y_i \right) &= {\rm var} \left( \sum_{i = 1}^n \beta_0 + \beta_1 X_i + \epsilon_i \right)\\ &= {\rm var} \left( \sum_{i = 1}^n \epsilon_i \right) = \sum_{i = 1}^n \sum_{j = 1}^n {\rm cov} (\epsilon_i, \epsilon_j)\\ &= \sum_{i = 1}^n {\rm cov} (\epsilon_i, \epsilon_i) = \sum_{i = 1}^n {\rm var} (\epsilon_i)\\ &= \sum_{i = 1}^n {\rm var} (\beta_0 + \beta_1 X_i + \epsilon_i) = \sum_{i = 1}^n {\rm var} (Y_i).\\ \end{align} The 4th equality holds as ${\rm cov} (\epsilon_i, \epsilon_j) = 0$ for $i \neq j$ by the independence of the $\epsilon_i$.

$\endgroup$
23
  • $\begingroup$ I think I got it! The book has suggested steps, and I was able to prove each step separately (I think). It's not as satisfying as just sitting down and grinding it out from this step, since I had to prove intermediate conclusions for it to help, but I think everything looks good. $\endgroup$
    – M T
    Commented Jul 14, 2013 at 0:25
  • $\begingroup$ See edit for the development of the suggested approach. $\endgroup$
    – QuantIbex
    Commented Jul 14, 2013 at 6:19
  • $\begingroup$ The variance of the sum equals the sum of the variances in this step: $$ {\rm Var} (\bar{Y}) = {\rm Var} \left(\frac{1}{n} \sum_{i = 1}^n Y_i \right) = \frac{1}{n^2} \sum_{i = 1}^n {\rm Var} (Y_i) $$ because since the $X_i$ are independent, this implies that the $Y_i$ are independent as well, right? $\endgroup$
    – M T
    Commented Jul 14, 2013 at 18:40
  • $\begingroup$ Also, you can factor out a constant from the covariance in this step: $$ \frac{1}{n} \frac{ 1 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } {\rm Cov} \left\{ \sum_{i = 1}^n Y_i, \sum_{j = 1}^n(x_j - \bar{x})Y_j \right\} $$ even though it's not in both elements because the formula for covariance is multiplicative, right? $\endgroup$
    – M T
    Commented Jul 14, 2013 at 18:42
  • 1
    $\begingroup$ @oort, in the numerator you have the sum of $n$ terms that are identical (and equal to $\sigma^2$), so the numerator is $n \sigma^2$. $\endgroup$
    – QuantIbex
    Commented Apr 7, 2016 at 14:40
1
$\begingroup$

I got it! Well, with help. I found the part of the book that gives steps to work through when proving the $Var \left( \hat{\beta}_0 \right)$ formula (thankfully it doesn't actually work them out, otherwise I'd be tempted to not actually do the proof). I proved each separate step, and I think it worked.

I'm using the book's notation, which is: $$ SST_x = \displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2, $$ and $u_i$ is the error term.

1) Show that $\hat{\beta}_1$ can be written as $\hat{\beta}_1 = \beta_1 + \displaystyle\sum\limits_{i=1}^n w_i u_i$ where $w_i = \frac{d_i}{SST_x}$ and $d_i = x_i - \bar{x}$.

This was easy because we know that

\begin{align} \hat{\beta}_1 &= \beta_1 + \frac{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) u_i}{SST_x} \\ &= \beta_1 + \displaystyle\sum\limits_{i=1}^n \frac{d_i}{SST_x} u_i \\ &= \beta_1 + \displaystyle\sum\limits_{i=1}^n w_i u_i \end{align}

2) Use part 1, along with $\displaystyle\sum\limits_{i=1}^n w_i = 0$ to show that $\hat{\beta_1}$ and $\bar{u}$ are uncorrelated, i.e. show that $E[(\hat{\beta_1}-\beta_1) \bar{u}] = 0$.

\begin{align} E[(\hat{\beta_1}-\beta_1) \bar{u}] &= E[\bar{u}\displaystyle\sum\limits_{i=1}^n w_i u_i] \\ &=\displaystyle\sum\limits_{i=1}^n E[w_i \bar{u} u_i] \\ &=\displaystyle\sum\limits_{i=1}^n w_i E[\bar{u} u_i] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E\left(u_i\displaystyle\sum\limits_{j=1}^n u_j\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E\left(u_i u_1\right) +\cdots + E(u_i u_j) + \cdots+ E\left(u_i u_n \right)\right] \\ \end{align}

and because the $u$ are i.i.d., $E(u_i u_j) = E(u_i) E(u_j)$ when $ j \neq i$.

When $j = i$, $E(u_i u_j) = E(u_i^2)$, so we have:

\begin{align} &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E(u_i) E(u_1) +\cdots + E(u_i^2) + \cdots + E(u_i) E(u_n)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E(u_i^2) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[Var(u_i) + E(u_i) E(u_i)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \sigma^2 \\ &= \frac{\sigma^2}{n}\displaystyle\sum\limits_{i=1}^n w_i \\ &= \frac{\sigma^2}{n \cdot SST_x}\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) \\ &= \frac{\sigma^2}{n \cdot SST_x} \left(0\right) &= 0 \end{align}

3) Show that $\hat{\beta_0}$ can be written as $\hat{\beta_0} = \beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1)$. This seemed pretty easy too:

\begin{align} \hat{\beta_0} &= \bar{y} - \hat{\beta_1} \bar{x} \\ &= (\beta_0 + \beta_1 \bar{x} + \bar{u}) - \hat{\beta_1} \bar{x} \\ &= \beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1). \end{align}

4) Use parts 2 and 3 to show that $Var(\hat{\beta_0}) = \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x}$: \begin{align} Var(\hat{\beta_0}) &= Var(\beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1)) \\ &= Var(\bar{u}) + (-\bar{x})^2 Var(\hat{\beta_1} - \beta_1) \\ &= \frac{\sigma^2}{n} + (\bar{x})^2 Var(\hat{\beta_1}) \\ &= \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x}. \end{align}

I believe this all works because since we provided that $\bar{u}$ and $\hat{\beta_1} - \beta_1$ are uncorrelated, the covariance between them is zero, so the variance of the sum is the sum of the variance. $\beta_0$ is just a constant, so it drops out, as does $\beta_1$ later in the calculations.

5) Use algebra and the fact that $\frac{SST_x}{n} = \frac{1}{n} \displaystyle\sum\limits_{i=1}^n x_i^2 - (\bar{x})^2$:

\begin{align} Var(\hat{\beta_0}) &= \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x} \\ &= \frac{\sigma^2 SST_x}{SST_x n} + \frac{\sigma^2 (\bar{x})^2}{SST_x} \\ &= \frac{\sigma^2}{SST_x} \left( \frac{1}{n} \displaystyle\sum\limits_{i=1}^n x_i^2 - (\bar{x})^2 \right) + \frac{\sigma^2 (\bar{x})^2}{SST_x} \\ &= \frac{\sigma^2 n^{-1} \displaystyle\sum\limits_{i=1}^n x_i^2}{SST_x} \end{align}

$\endgroup$
22
  • $\begingroup$ There might be a typo in point 1; I think ${\rm var(\hat{\beta})}$ should read $\hat{\beta}$. $\endgroup$
    – QuantIbex
    Commented Jul 15, 2013 at 22:10
  • $\begingroup$ You might want to clarify notations, and specify what $u_i$ and ${\rm SST}_x$ are. $\endgroup$
    – QuantIbex
    Commented Jul 15, 2013 at 22:13
  • $\begingroup$ $u_i$ is the error term and $SST_x$ is the total sum of squares for $x$ (defined in the edit). $\endgroup$
    – M T
    Commented Jul 15, 2013 at 22:37
  • 1
    $\begingroup$ In point 1, the term $\beta_1$ is missing in the last two lines. $\endgroup$
    – QuantIbex
    Commented Jul 16, 2013 at 6:06
  • 1
    $\begingroup$ In point 2, you can't take $\bar{u}$ out of the expectation, it's not a constant. $\endgroup$
    – QuantIbex
    Commented Jul 16, 2013 at 6:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.