Finding Variance for Simple Linear Regression Coefficients

Question

Suppose we have the simple linear model $$\mathbf{y} = \beta_0 \mathbf{1} + \beta_1 \mathbf{x} + \boldsymbol{\epsilon},$$ with $\mathrm{E}[\boldsymbol{\epsilon}] = \mathbf{0}$ and $\mathrm{Var}[\boldsymbol{\epsilon}] = \sigma^2 \mathbf{I}$. The least squares estimator for $\beta_1$ is $$\hat{\beta}_1 = \frac{(\mathbf{x} - \bar{x}\mathbf{1})^T(\mathbf{y} - \bar{y}\mathbf{1})}{\|\mathbf{x} - \bar{x}\mathbf{1}\|^2}.$$ For this problem, we are treating $\mathbf{y}$ as a random vector (and so $\bar{y}$ is a random variable), so we have $\mathrm{E}[\mathbf{y}] = \beta_0 \mathbf{1} + \beta_1 \mathbf{x}$ (so for each response we have $\mathrm{E}[y_i] = \beta_0 + \beta_1 x_i$), and $\mathrm{E}[\bar{y}] = \beta_0 + \beta_1 \bar{x}$. In addition, we have $\mathrm{Var}[y_i] = \sigma^2$ for all $i$ and $\mathrm{Var}[\bar{y}] = \sigma^2 / n$.

Assuming that $\mathrm{E}[\hat{\beta}_1] = \beta_1$, we want to show that $$\mathrm{Var}[\hat{\beta}_1] = \frac{\sigma^2}{\|\mathbf{x} - \bar{x}\mathbf{1}\|^2}.$$ To start, it's helpful to re-write $\hat{\beta}_1$ as $$\hat{\beta}_1 = \frac{1}{\|\mathbf{x} - \bar{x}\mathbf{1}\|^2} \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}).$$ Again, we emphasize that the only sources of randomness in this are $y_i$ and $\bar{y}$ (which are in turn random because of $\epsilon_i$.)

Here is what I did: we have $$ \begin{align*} \mathrm{Var}[\hat{\beta}_1] &= \mathrm{Var} \left[\frac{1}{\|\mathbf{x} - \bar{x}\mathbf{1}\|^2} \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) \right] \\ &= \frac{1}{\|\mathbf{x} - \bar{x}\mathbf{1}\|^4} \cdot \mathrm{Var} \left[ \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})\right] \\ &= \frac{1}{\|\mathbf{x} - \bar{x}\mathbf{1}\|^4} \sum_{i=1}^n \mathrm{Var} \Big[ (x_i - \bar{x})(y_i - \bar{y}) \Big] \\ &= \frac{1}{\|\mathbf{x} - \bar{x}\mathbf{1}\|^4} \sum_{i=1}^n (x_i - \bar{x})^2 \cdot \mathrm{Var} \big[ (y_i - \bar{y}) \big] \\ &= \frac{1}{\|\mathbf{x} - \bar{x}\mathbf{1}\|^4} \sum_{i=1}^n (x_i - \bar{x})^2 \cdot \Big( \mathrm{Var}[y_i] + \mathrm{Var}[\bar{y}]\Big) \\ &= \frac{1}{\|\mathbf{x} - \bar{x}\mathbf{1}\|^4} \sum_{i=1}^n (x_i - \bar{x})^2 \cdot \Big( \sigma^2 + \sigma^2 / n \Big) \\ &= \frac{n+1}{n} \cdot \frac{\sigma^2}{\|\mathbf{x} - \bar{x}\mathbf{1}\|^2}. \end{align*} $$ So I'm close to the answer, but it's off by a factor of $(n+1)/n$.

When looking to see what others did, it seems that the trick is to get rid of the $\bar{y}$ in the equation altogether, since $$ \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) = \sum_{i=1}^n (x_i - \bar{x})y_i + \underbrace{\sum_{i=1}^n (x_i - \bar{x}) \bar{y}}_{= ~0} = \sum_{i=1}^n (x_i - \bar{x}) y_i. $$ Once this is done, the extra variance added by $\bar{y}$ is gone, and we get the correct answer. I get that, but I don't understand why the way I did it is incorrect, and leaving it in gives the incorrect answer.

What is wrong with leaving it in? Did I make a mistake with my calculations?

Answer 1

The variance of a sum of random variables only equals the sum of their variances if the random variables are uncorrelated. If they are correlated a covariance term needs to be included:

$ var(X + Y) = var(X) + var(Y) + 2 cov(X, Y) $

where $X$ and $Y$ are random variables. This follows from the fact that the covariance is linear in each of its arguments ($\alpha$ and $\beta$ are non-random scalars):

$ v\alpha r(\alpha X + \beta Y) $

$ = cov(\alpha X + \beta Y, \alpha X + \beta Y)$

$ = \alpha^2 var(X) + \beta^2 var(Y) + 2 \alpha \beta cov(X, Y)$

See https://en.wikipedia.org/wiki/Variance#Sum_of_correlated_variables

Expanding expression directly

To expand out the expression using the above we define $S_{xx} = \Sigma_{i=1}^n (x_i-\bar{x})^2$. In the second last line we use the facts that the $y_i$ values are uncorrelated for distinct $i$ and $cov$ is linear in each of its arguments so $ cov(y_i, \bar{y}) = cov(y_i, y_i/n) = \sigma^2/n $ .

$ var(\hat{\beta}) $

$ = var(\Sigma_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) / S_{xx}) $

$ = \Sigma_{i=1}^n var((x_i - \bar{x})y_i/S_{xx}) + \Sigma_{i=1}^n var((x_i - \bar{x})\bar{y}/S_{xx}) + \Sigma_{i=1}^n \Sigma_{j=1}^n cov((x_i - \bar{x})y_i/S_{xx}, -(x_i - \bar{x})\bar{y} / S_{xx})$

$ = \sigma^2 / S_{xx} + \sigma^2 / S_{xx} - \sigma^2 / S_{xx} $

$ = \sigma^2 / S_{xx} $

Expressing $\hat{\beta}$ as linear function of $y$

Although I gather from the question that you already understand this it is messy expanding the variance using the expression we have just shown so let us just repeat the approach of avoiding the complex calculations and instead express $\hat{\beta}$ as a linear function of $y$.

In the third line the fact is used that the vector whose components are $x_i-\bar{x}$ is orthogonal to the space of constant vectors and in the last line we use the fact that the components of $y$ are uncorrelated so that the variance of the sum equals the sum of the variances.

$ var(\hat{\beta}) $

$ = var(\Sigma_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) / S_{xx}) $

$ = var(\Sigma_{i=1}^n (x_i - \bar{x})y_i / S_{xx}) $

$ = \sigma^2 / S_{xx} $