0
$\begingroup$

I am currently reading through slides from Georgia Tech on linear regression and came across a section that has confused me. It states for $$ y_i=\beta_0+\beta_1x_i+\epsilon_i $$ where $\epsilon_i \sim N(0,\sigma^2)$ and $$ \hat{\beta}_1=\frac{\sum_{i=1}^n(x_i-\bar{x})y_i}{\sum_{i=1}^n(x_i-\bar{x})^2} $$ the covariance is $$ \begin{align*} Cov(\bar{y},\hat{\beta}_1)&=\frac{1}{\sum_{i=1}^n(x_i-\bar{x})^2}Cov\Big(\bar{y},\sum_{i=1}^n(x_i-\bar{x})y_i\Big) \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})}{\sum_{i=1}^n(x_i-\bar{x})^2}Cov\Big(\bar{y},y_i\Big) \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})}{\sum_{i=1}^n(x_i-\bar{x})^2}\frac{\sigma^2}{n} \\ &= 0 \end{align*} $$ Now, I assume that it becomes 0 from the $\sum_{i=1}^n(x_i-\bar{x})=0$ term. However, what is confusing me is how we can pull out the $\sum_{i=1}^n(x_i-\bar{x})$ term from the $Cov\Big(\bar{y},\sum_{i=1}^n(x_i-\bar{x})y_i\Big)$ since the $y_i$ is part of the summation and not constant for all $i$ values (or so I thought).

$\endgroup$
  • 1
    $\begingroup$ Under i.i.d normality assumption on the residuals, $\bar y$ and $\hat\beta_1$ are in fact independently distributed. $\endgroup$ – StubbornAtom Oct 16 '19 at 11:15
2
$\begingroup$

You don't pull out $\sum_{i=1}^n (x_i - \bar{x})$. Instead, you pull out $(x_i - \bar{x})$ $n$ times. Throughout, the $(x_i)_{1 \leq i \leq n}$ sequence is taken to be non-random. The way the original argument is written up is a little confusing. The following may be easier to understand: $$ \begin{align*} Cov(\bar{y},\hat{\beta}_1)&=\frac{1}{\sum_{j=1}^n(x_j-\bar{x})^2}Cov\Big(\bar{y},\sum_{i=1}^n(x_i-\bar{x})y_i\Big) \\ &=\frac{1}{\sum_{j=1}^n(x_j-\bar{x})^2}Cov\Big(\bar{y},(x_1-\bar{x})y_1 + \dotsm + (x_n-\bar{x})y_n \Big) \\ &=\frac{(x_1-\bar{x})Cov(\bar{y}, y_1) + \dotsm + (x_n-\bar{x})Cov(\bar{y}, y_n)}{\sum_{j=1}^n(x_j-\bar{x})^2} \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})Cov\Big(\bar{y},y_i\Big)}{\sum_{j=1}^n(x_j-\bar{x})^2} \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})}{\sum_{j=1}^n(x_j-\bar{x})^2}\frac{\sigma^2}{n} \\ &= 0. \end{align*} $$ To evaluate $Cov(\bar{y}, y_i)$ note that $Cov(y_j, y_i)=0$ for $j \neq i$ and write \begin{align*} Cov(\bar{y}, y_i) =& n^{-1} Cov(y_1 + \dotsm + y_n, y_i) \\ =& n^{-1} \Big(0 + \dotsm + 0 + Cov(y_i, y_i) + 0 \dotsm + 0\Big) \\ =& n^{-1} Var(y_i) = n^{-1} Var(\beta_0 + \beta_1 x_i + \epsilon_i) = n^{-1} Var(\epsilon_i) = \sigma^2/n. \end{align*}

$\endgroup$
  • $\begingroup$ @AndreasDzemski So we treat $(x_i-\bar{x})$ as a constant? If that's the case, then why don't we still keep a $\sum_{i=1}^ny_i$ in the $Cov$ function as well? Making it $Cov(\bar{y},\sum_{i=1}^ny_i)$ $\endgroup$ – strwars Oct 15 '19 at 20:07
  • 1
    $\begingroup$ Yes the $x_i$ variables are treated as non-random. $Cov(\bar y \sum_i (x_i -\bar x)y_i) = Cov(\bar y, (x_1 - \bar x)y_1 + ... + (x_N - \bar x)y_N) = (x_1-\bar x)Cov(\bar y,y_1) + ... + (x_N - \bar x) Cov(\bar y,y_N) = \sum_{i}( x_i - \bar x) Cov(\bar y,y_i) = \sum_{i}( x_i - \bar x) Cov(\bar \epsilon,\epsilon_i ) = (1/n) \sum_{i}( x_i - \bar x) Cov(\epsilon_i,\epsilon_i)$ $\endgroup$ – Stop Closing Questions Fast Oct 15 '19 at 20:17
  • $\begingroup$ @JesperHybel thank you for the explanation, it has cleared a lot up! I assume the $Cov(\bar{\epsilon_i},\epsilon_i)$ is converted by $Cov(\bar{\epsilon_i},\epsilon_i)=Cov\Big(\frac{1}{n}(\epsilon_1+\epsilon_2+...+\epsilon_n),\epsilon_i\Big)=\frac{1}{n}Cov(\sum_{i=1}^n\epsilon_i,\epsilon_i)$ and $\sum_{i=1}^n(x_i+\bar{x})=0$ makes the whole thing 0. $\endgroup$ – strwars Oct 15 '19 at 22:43
  • 1
    $\begingroup$ Yes and independence is used to get covariance with the sum of errors and a single error to be simply variance of the single error. $\endgroup$ – Stop Closing Questions Fast Oct 16 '19 at 5:53
  • 1
    $\begingroup$ I added Jesper's clarifying comments to the answer. $\endgroup$ – Andreas Dzemski Oct 16 '19 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.