I am currently reading through slides from Georgia Tech on linear regression and came across a section that has confused me. It states for $$ y_i=\beta_0+\beta_1x_i+\epsilon_i $$ where $\epsilon_i \sim N(0,\sigma^2)$ and $$ \hat{\beta}_1=\frac{\sum_{i=1}^n(x_i-\bar{x})y_i}{\sum_{i=1}^n(x_i-\bar{x})^2} $$ the covariance is $$ \begin{align*} Cov(\bar{y},\hat{\beta}_1)&=\frac{1}{\sum_{i=1}^n(x_i-\bar{x})^2}Cov\Big(\bar{y},\sum_{i=1}^n(x_i-\bar{x})y_i\Big) \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})}{\sum_{i=1}^n(x_i-\bar{x})^2}Cov\Big(\bar{y},y_i\Big) \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})}{\sum_{i=1}^n(x_i-\bar{x})^2}\frac{\sigma^2}{n} \\ &= 0 \end{align*} $$ Now, I assume that it becomes 0 from the $\sum_{i=1}^n(x_i-\bar{x})=0$ term. However, what is confusing me is how we can pull out the $\sum_{i=1}^n(x_i-\bar{x})$ term from the $Cov\Big(\bar{y},\sum_{i=1}^n(x_i-\bar{x})y_i\Big)$ since the $y_i$ is part of the summation and not constant for all $i$ values (or so I thought).
1 Answer
You don't pull out $\sum_{i=1}^n (x_i - \bar{x})$. Instead, you pull out $(x_i - \bar{x})$ $n$ times. Throughout, the $(x_i)_{1 \leq i \leq n}$ sequence is taken to be non-random. The way the original argument is written up is a little confusing. The following may be easier to understand: $$ \begin{align*} Cov(\bar{y},\hat{\beta}_1)&=\frac{1}{\sum_{j=1}^n(x_j-\bar{x})^2}Cov\Big(\bar{y},\sum_{i=1}^n(x_i-\bar{x})y_i\Big) \\ &=\frac{1}{\sum_{j=1}^n(x_j-\bar{x})^2}Cov\Big(\bar{y},(x_1-\bar{x})y_1 + \dotsm + (x_n-\bar{x})y_n \Big) \\ &=\frac{(x_1-\bar{x})Cov(\bar{y}, y_1) + \dotsm + (x_n-\bar{x})Cov(\bar{y}, y_n)}{\sum_{j=1}^n(x_j-\bar{x})^2} \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})Cov\Big(\bar{y},y_i\Big)}{\sum_{j=1}^n(x_j-\bar{x})^2} \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})}{\sum_{j=1}^n(x_j-\bar{x})^2}\frac{\sigma^2}{n} \\ &= 0. \end{align*} $$ To evaluate $Cov(\bar{y}, y_i)$ note that $Cov(y_j, y_i)=0$ for $j \neq i$ and write \begin{align*} Cov(\bar{y}, y_i) =& n^{-1} Cov(y_1 + \dotsm + y_n, y_i) \\ =& n^{-1} \Big(0 + \dotsm + 0 + Cov(y_i, y_i) + 0 \dotsm + 0\Big) \\ =& n^{-1} Var(y_i) = n^{-1} Var(\beta_0 + \beta_1 x_i + \epsilon_i) = n^{-1} Var(\epsilon_i) = \sigma^2/n. \end{align*}
-
1$\begingroup$ @AndreasDzemski So we treat $(x_i-\bar{x})$ as a constant? If that's the case, then why don't we still keep a $\sum_{i=1}^ny_i$ in the $Cov$ function as well? Making it $Cov(\bar{y},\sum_{i=1}^ny_i)$ $\endgroup$– strwarsOct 15, 2019 at 20:07
-
1$\begingroup$ Yes the $x_i$ variables are treated as non-random. $Cov(\bar y \sum_i (x_i -\bar x)y_i) = Cov(\bar y, (x_1 - \bar x)y_1 + ... + (x_N - \bar x)y_N) = (x_1-\bar x)Cov(\bar y,y_1) + ... + (x_N - \bar x) Cov(\bar y,y_N) = \sum_{i}( x_i - \bar x) Cov(\bar y,y_i) = \sum_{i}( x_i - \bar x) Cov(\bar \epsilon,\epsilon_i ) = (1/n) \sum_{i}( x_i - \bar x) Cov(\epsilon_i,\epsilon_i)$ $\endgroup$ Oct 15, 2019 at 20:17
-
$\begingroup$ @JesperHybel thank you for the explanation, it has cleared a lot up! I assume the $Cov(\bar{\epsilon_i},\epsilon_i)$ is converted by $Cov(\bar{\epsilon_i},\epsilon_i)=Cov\Big(\frac{1}{n}(\epsilon_1+\epsilon_2+...+\epsilon_n),\epsilon_i\Big)=\frac{1}{n}Cov(\sum_{i=1}^n\epsilon_i,\epsilon_i)$ and $\sum_{i=1}^n(x_i+\bar{x})=0$ makes the whole thing 0. $\endgroup$– strwarsOct 15, 2019 at 22:43
-
1$\begingroup$ Yes and independence is used to get covariance with the sum of errors and a single error to be simply variance of the single error. $\endgroup$ Oct 16, 2019 at 5:53
-
1$\begingroup$ I added Jesper's clarifying comments to the answer. $\endgroup$ Oct 16, 2019 at 9:40