# How do I calculate the variance of the OLS estimator $\beta_0$, conditional on $x_1, \ldots , x_n$?

I know that $$\hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x}$$ and this is how far I got when I calculated the variance:

\begin{align*} Var(\hat{\beta_0}) &= Var(\bar{y} - \hat{\beta_1}\bar{x}) \\ &= Var((-\bar{x})\hat{\beta_1}+\bar{y}) \\ &= Var((-\bar{x})\hat{\beta_1})+Var(\bar{y}) \\ &= (-\bar{x})^2 Var(\hat{\beta_1}) + 0 \\ &= (\bar{x})^2 Var(\hat{\beta_1}) + 0 \\ &= \frac{\sigma^2 (\bar{x})^2}{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2} \end{align*}

but that's far as I got. The final formula I'm trying to calculate is

\begin{align*} Var(\hat{\beta_0}) &= \frac{\sigma^2 n^{-1}\displaystyle\sum\limits_{i=1}^n x_i^2}{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2} \end{align*}

I'm not sure how to get $$(\bar{x})^2 = \frac{1}{n}\displaystyle\sum\limits_{i=1}^n x_i^2$$ assuming my math is correct up to there.

Is this the right path?

\begin{align} (\bar{x})^2 &= \left(\frac{1}{n}\displaystyle\sum\limits_{i=1}^n x_i\right)^2 \\ &= \frac{1}{n^2} \left(\displaystyle\sum\limits_{i=1}^n x_i\right)^2 \end{align}

I'm sure it's simple, so the answer can wait for a bit if someone has a hint to push me in the right direction.

• This is not the right path. The 4th equation doesn't hold. For example, with $x_1=−1$, $x_2=0$, and $x_3=1$, the left term is zero, whilst the right term is $2/3$. The problem comes from the step where you split the variance (3rd line of second equation). See why? – QuantIbex Jul 12 '13 at 23:55
• Hint towards Quantlbex point: variance is not a linear function. It violates both additivity and scalar multiplication. – David Marx Jul 13 '13 at 1:53
• @DavidMarx That step should be $$=Var((-\bar{x})\hat{\beta_1}+\bar{y})=(\bar{x})^2Var(\hat{\beta_1})+\bar{y}$$, I think, and then once I substitute in for $\hat{\beta_1}$ and $\bar{y}$ (not sure what to do for this but I'll think about it more), that should put me on the right path I hope. – M T Jul 13 '13 at 3:53
• This is not correct. Think about the condition required for the variance of a sum to be equal to the sum of the variances. – QuantIbex Jul 13 '13 at 10:29
• No, $\bar{y}$ is random since $y_i = \beta_0 + \beta_1 x_i + \epsilon$, where $\epsilon$ denotes the (random) noise. But OK, my previous comment was maybe misleading. Also, ${\rm Var}(aX + b)= a^2{\rm Var}(X)$, if $a$ and $b$ denote constants. – QuantIbex Jul 13 '13 at 19:38

This is a self-study question, so I provide hints that will hopefully help to find the solution, and I'll edit the answer based on your feedbacks/progress.

The parameter estimates that minimize the sum of squares are \begin{align} \hat{\beta}_0 &= \bar{y} - \hat{\beta}_1 \bar{x} , \\ \hat{\beta}_1 &= \frac{ \sum_{i = 1}^n(x_i - \bar{x})y_i }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } . \end{align} To get the variance of $\hat{\beta}_0$, start from its expression and substitute the expression of $\hat{\beta}_1$, and do the algebra $${\rm Var}(\hat{\beta}_0) = {\rm Var} (\bar{Y} - \hat{\beta}_1 \bar{x}) = \ldots$$

Edit:
We have \begin{align} {\rm Var}(\hat{\beta}_0) &= {\rm Var} (\bar{Y} - \hat{\beta}_1 \bar{x}) \\ &= {\rm Var} (\bar{Y}) + (\bar{x})^2 {\rm Var} (\hat{\beta}_1) - 2 \bar{x} {\rm Cov} (\bar{Y}, \hat{\beta}_1). \end{align} The two variance terms are $${\rm Var} (\bar{Y}) = {\rm Var} \left(\frac{1}{n} \sum_{i = 1}^n Y_i \right) = \frac{1}{n^2} \sum_{i = 1}^n {\rm Var} (Y_i) = \frac{\sigma^2}{n},$$ and \begin{align} {\rm Var} (\hat{\beta}_1) &= \frac{ 1 }{ \left[\sum_{i = 1}^n(x_i - \bar{x})^2 \right]^2 } \sum_{i = 1}^n(x_i - \bar{x})^2 {\rm Var} (Y_i) \\ &= \frac{ \sigma^2 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } , \end{align} and the covariance term is \begin{align} {\rm Cov} (\bar{Y}, \hat{\beta}_1) &= {\rm Cov} \left\{ \frac{1}{n} \sum_{i = 1}^n Y_i, \frac{ \sum_{j = 1}^n(x_j - \bar{x})Y_j }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } \right \} \\ &= \frac{1}{n} \frac{ 1 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } {\rm Cov} \left\{ \sum_{i = 1}^n Y_i, \sum_{j = 1}^n(x_j - \bar{x})Y_j \right\} \\ &= \frac{ 1 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \sum_{i = 1}^n (x_j - \bar{x}) \sum_{j = 1}^n {\rm Cov}(Y_i, Y_j) \\ &= \frac{ 1 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \sum_{i = 1}^n (x_j - \bar{x}) \sigma^2 \\ &= 0 \end{align} since $\sum_{i = 1}^n (x_j - \bar{x})=0$.
And since $$\sum_{i = 1}^n(x_i - \bar{x})^2 = \sum_{i = 1}^n x_i^2 - 2 \bar{x} \sum_{i = 1}^n x_i + \sum_{i = 1}^n \bar{x}^2 = \sum_{i = 1}^n x_i^2 - n \bar{x}^2,$$ we have \begin{align} {\rm Var}(\hat{\beta}_0) &= \frac{\sigma^2}{n} + \frac{ \sigma^2 \bar{x}^2}{ \sum_{i = 1}^n(x_i - \bar{x})^2 } \\ &= \frac{\sigma^2 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \left\{ \sum_{i = 1}^n(x_i - \bar{x})^2 + n \bar{x}^2 \right\} \\ &= \frac{\sigma^2 \sum_{i = 1}^n x_i^2}{ n \sum_{i = 1}^n(x_i - \bar{x})^2 }. \end{align}

Edit 2

Why do we have ${\rm var} ( \sum_{i = 1}^n Y_i) = \sum_{i = 1}^n {\rm Var} (Y_i)$?

The assumed model is $Y_i = \beta_0 + \beta_1 X_i + \epsilon_i$, where the $\epsilon_i$ are independant and identically distributed random variables with ${\rm E}(\epsilon_i) = 0$ and ${\rm var}(\epsilon_i) = \sigma^2$.

Once we have a sample, the $X_i$ are known, the only random terms are the $\epsilon_i$. Recalling that for a random variable $Z$ and a constant $a$, we have ${\rm var}(a+Z) = {\rm var}(Z)$. Thus, \begin{align} {\rm var} \left( \sum_{i = 1}^n Y_i \right) &= {\rm var} \left( \sum_{i = 1}^n \beta_0 + \beta_1 X_i + \epsilon_i \right)\\ &= {\rm var} \left( \sum_{i = 1}^n \epsilon_i \right) = \sum_{i = 1}^n \sum_{j = 1}^n {\rm cov} (\epsilon_i, \epsilon_j)\\ &= \sum_{i = 1}^n {\rm cov} (\epsilon_i, \epsilon_i) = \sum_{i = 1}^n {\rm var} (\epsilon_i)\\ &= \sum_{i = 1}^n {\rm var} (\beta_0 + \beta_1 X_i + \epsilon_i) = \sum_{i = 1}^n {\rm var} (Y_i).\\ \end{align} The 4th equality holds as ${\rm cov} (\epsilon_i, \epsilon_j) = 0$ for $i \neq j$ by the independence of the $\epsilon_i$.

• I think I got it! The book has suggested steps, and I was able to prove each step separately (I think). It's not as satisfying as just sitting down and grinding it out from this step, since I had to prove intermediate conclusions for it to help, but I think everything looks good. – M T Jul 14 '13 at 0:25
• See edit for the development of the suggested approach. – QuantIbex Jul 14 '13 at 6:19
• The variance of the sum equals the sum of the variances in this step: $${\rm Var} (\bar{Y}) = {\rm Var} \left(\frac{1}{n} \sum_{i = 1}^n Y_i \right) = \frac{1}{n^2} \sum_{i = 1}^n {\rm Var} (Y_i)$$ because since the $X_i$ are independent, this implies that the $Y_i$ are independent as well, right? – M T Jul 14 '13 at 18:40
• Also, you can factor out a constant from the covariance in this step: $$\frac{1}{n} \frac{ 1 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } {\rm Cov} \left\{ \sum_{i = 1}^n Y_i, \sum_{j = 1}^n(x_j - \bar{x})Y_j \right\}$$ even though it's not in both elements because the formula for covariance is multiplicative, right? – M T Jul 14 '13 at 18:42
• @oort, in the numerator you have the sum of $n$ terms that are identical (and equal to $\sigma^2$), so the numerator is $n \sigma^2$. – QuantIbex Apr 7 '16 at 14:40

I got it! Well, with help. I found the part of the book that gives steps to work through when proving the $Var \left( \hat{\beta}_0 \right)$ formula (thankfully it doesn't actually work them out, otherwise I'd be tempted to not actually do the proof). I proved each separate step, and I think it worked.

I'm using the book's notation, which is: $$SST_x = \displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2,$$ and $u_i$ is the error term.

1) Show that $\hat{\beta}_1$ can be written as $\hat{\beta}_1 = \beta_1 + \displaystyle\sum\limits_{i=1}^n w_i u_i$ where $w_i = \frac{d_i}{SST_x}$ and $d_i = x_i - \bar{x}$.

This was easy because we know that

\begin{align} \hat{\beta}_1 &= \beta_1 + \frac{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) u_i}{SST_x} \\ &= \beta_1 + \displaystyle\sum\limits_{i=1}^n \frac{d_i}{SST_x} u_i \\ &= \beta_1 + \displaystyle\sum\limits_{i=1}^n w_i u_i \end{align}

2) Use part 1, along with $\displaystyle\sum\limits_{i=1}^n w_i = 0$ to show that $\hat{\beta_1}$ and $\bar{u}$ are uncorrelated, i.e. show that $E[(\hat{\beta_1}-\beta_1) \bar{u}] = 0$.

\begin{align} E[(\hat{\beta_1}-\beta_1) \bar{u}] &= E[\bar{u}\displaystyle\sum\limits_{i=1}^n w_i u_i] \\ &=\displaystyle\sum\limits_{i=1}^n E[w_i \bar{u} u_i] \\ &=\displaystyle\sum\limits_{i=1}^n w_i E[\bar{u} u_i] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E\left(u_i\displaystyle\sum\limits_{j=1}^n u_j\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E\left(u_i u_1\right) +\cdots + E(u_i u_j) + \cdots+ E\left(u_i u_n \right)\right] \\ \end{align}

and because the $u$ are i.i.d., $E(u_i u_j) = E(u_i) E(u_j)$ when $j \neq i$.

When $j = i$, $E(u_i u_j) = E(u_i^2)$, so we have:

\begin{align} &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E(u_i) E(u_1) +\cdots + E(u_i^2) + \cdots + E(u_i) E(u_n)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E(u_i^2) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[Var(u_i) + E(u_i) E(u_i)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \sigma^2 \\ &= \frac{\sigma^2}{n}\displaystyle\sum\limits_{i=1}^n w_i \\ &= \frac{\sigma^2}{n \cdot SST_x}\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) \\ &= \frac{\sigma^2}{n \cdot SST_x} \left(0\right) &= 0 \end{align}

3) Show that $\hat{\beta_0}$ can be written as $\hat{\beta_0} = \beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1)$. This seemed pretty easy too:

\begin{align} \hat{\beta_0} &= \bar{y} - \hat{\beta_1} \bar{x} \\ &= (\beta_0 + \beta_1 \bar{x} + \bar{u}) - \hat{\beta_1} \bar{x} \\ &= \beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1). \end{align}

4) Use parts 2 and 3 to show that $Var(\hat{\beta_0}) = \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x}$: \begin{align} Var(\hat{\beta_0}) &= Var(\beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1)) \\ &= Var(\bar{u}) + (-\bar{x})^2 Var(\hat{\beta_1} - \beta_1) \\ &= \frac{\sigma^2}{n} + (\bar{x})^2 Var(\hat{\beta_1}) \\ &= \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x}. \end{align}

I believe this all works because since we provided that $\bar{u}$ and $\hat{\beta_1} - \beta_1$ are uncorrelated, the covariance between them is zero, so the variance of the sum is the sum of the variance. $\beta_0$ is just a constant, so it drops out, as does $\beta_1$ later in the calculations.

5) Use algebra and the fact that $\frac{SST_x}{n} = \frac{1}{n} \displaystyle\sum\limits_{i=1}^n x_i^2 - (\bar{x})^2$:

\begin{align} Var(\hat{\beta_0}) &= \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x} \\ &= \frac{\sigma^2 SST_x}{SST_x n} + \frac{\sigma^2 (\bar{x})^2}{SST_x} \\ &= \frac{\sigma^2}{SST_x} \left( \frac{1}{n} \displaystyle\sum\limits_{i=1}^n x_i^2 - (\bar{x})^2 \right) + \frac{\sigma^2 (\bar{x})^2}{SST_x} \\ &= \frac{\sigma^2 n^{-1} \displaystyle\sum\limits_{i=1}^n x_i^2}{SST_x} \end{align}

• There might be a typo in point 1; I think ${\rm var(\hat{\beta})}$ should read $\hat{\beta}$. – QuantIbex Jul 15 '13 at 22:10
• You might want to clarify notations, and specify what $u_i$ and ${\rm SST}_x$ are. – QuantIbex Jul 15 '13 at 22:13
• $u_i$ is the error term and $SST_x$ is the total sum of squares for $x$ (defined in the edit). – M T Jul 15 '13 at 22:37
• In point 1, the term $\beta_1$ is missing in the last two lines. – QuantIbex Jul 16 '13 at 6:06
• In point 2, you can't take $\bar{u}$ out of the expectation, it's not a constant. – QuantIbex Jul 16 '13 at 6:07