0
$\begingroup$

I am currently reading through slides from Georgia Tech on linear regression and came across a section that has confused me. It states for $$ y_i=\beta_0+\beta_1x_i+\epsilon_i $$ where $\epsilon_i \sim N(0,\sigma^2)$ and $$ \hat{\beta}_1=\frac{\sum_{i=1}^n(x_i-\bar{x})y_i}{\sum_{i=1}^n(x_i-\bar{x})^2} $$ the covariance is $$ \begin{align*} Cov(\bar{y},\hat{\beta}_1)&=\frac{1}{\sum_{i=1}^n(x_i-\bar{x})^2}Cov\Big(\bar{y},\sum_{i=1}^n(x_i-\bar{x})y_i\Big) \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})}{\sum_{i=1}^n(x_i-\bar{x})^2}Cov\Big(\bar{y},y_i\Big) \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})}{\sum_{i=1}^n(x_i-\bar{x})^2}\frac{\sigma^2}{n} \\ &= 0 \end{align*} $$ Now, I assume that it becomes 0 from the $\sum_{i=1}^n(x_i-\bar{x})=0$ term. However, what is confusing me is how we can pull out the $\sum_{i=1}^n(x_i-\bar{x})$ term from the $Cov\Big(\bar{y},\sum_{i=1}^n(x_i-\bar{x})y_i\Big)$ since the $y_i$ is part of the summation and not constant for all $i$ values (or so I thought).

$\endgroup$
0

1 Answer 1

3
$\begingroup$

You don't pull out $\sum_{i=1}^n (x_i - \bar{x})$. Instead, you pull out $(x_i - \bar{x})$ $n$ times. Throughout, the $(x_i)_{1 \leq i \leq n}$ sequence is taken to be non-random. The way the original argument is written up is a little confusing. The following may be easier to understand: $$ \begin{align*} Cov(\bar{y},\hat{\beta}_1)&=\frac{1}{\sum_{j=1}^n(x_j-\bar{x})^2}Cov\Big(\bar{y},\sum_{i=1}^n(x_i-\bar{x})y_i\Big) \\ &=\frac{1}{\sum_{j=1}^n(x_j-\bar{x})^2}Cov\Big(\bar{y},(x_1-\bar{x})y_1 + \dotsm + (x_n-\bar{x})y_n \Big) \\ &=\frac{(x_1-\bar{x})Cov(\bar{y}, y_1) + \dotsm + (x_n-\bar{x})Cov(\bar{y}, y_n)}{\sum_{j=1}^n(x_j-\bar{x})^2} \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})Cov\Big(\bar{y},y_i\Big)}{\sum_{j=1}^n(x_j-\bar{x})^2} \\ &=\frac{\sum_{i=1}^n(x_i-\bar{x})}{\sum_{j=1}^n(x_j-\bar{x})^2}\frac{\sigma^2}{n} \\ &= 0. \end{align*} $$ To evaluate $Cov(\bar{y}, y_i)$ note that $Cov(y_j, y_i)=0$ for $j \neq i$ and write \begin{align*} Cov(\bar{y}, y_i) =& n^{-1} Cov(y_1 + \dotsm + y_n, y_i) \\ =& n^{-1} \Big(0 + \dotsm + 0 + Cov(y_i, y_i) + 0 \dotsm + 0\Big) \\ =& n^{-1} Var(y_i) = n^{-1} Var(\beta_0 + \beta_1 x_i + \epsilon_i) = n^{-1} Var(\epsilon_i) = \sigma^2/n. \end{align*}

$\endgroup$
5
  • 1
    $\begingroup$ @AndreasDzemski So we treat $(x_i-\bar{x})$ as a constant? If that's the case, then why don't we still keep a $\sum_{i=1}^ny_i$ in the $Cov$ function as well? Making it $Cov(\bar{y},\sum_{i=1}^ny_i)$ $\endgroup$
    – strwars
    Oct 15, 2019 at 20:07
  • 1
    $\begingroup$ Yes the $x_i$ variables are treated as non-random. $Cov(\bar y \sum_i (x_i -\bar x)y_i) = Cov(\bar y, (x_1 - \bar x)y_1 + ... + (x_N - \bar x)y_N) = (x_1-\bar x)Cov(\bar y,y_1) + ... + (x_N - \bar x) Cov(\bar y,y_N) = \sum_{i}( x_i - \bar x) Cov(\bar y,y_i) = \sum_{i}( x_i - \bar x) Cov(\bar \epsilon,\epsilon_i ) = (1/n) \sum_{i}( x_i - \bar x) Cov(\epsilon_i,\epsilon_i)$ $\endgroup$ Oct 15, 2019 at 20:17
  • $\begingroup$ @JesperHybel thank you for the explanation, it has cleared a lot up! I assume the $Cov(\bar{\epsilon_i},\epsilon_i)$ is converted by $Cov(\bar{\epsilon_i},\epsilon_i)=Cov\Big(\frac{1}{n}(\epsilon_1+\epsilon_2+...+\epsilon_n),\epsilon_i\Big)=\frac{1}{n}Cov(\sum_{i=1}^n\epsilon_i,\epsilon_i)$ and $\sum_{i=1}^n(x_i+\bar{x})=0$ makes the whole thing 0. $\endgroup$
    – strwars
    Oct 15, 2019 at 22:43
  • 1
    $\begingroup$ Yes and independence is used to get covariance with the sum of errors and a single error to be simply variance of the single error. $\endgroup$ Oct 16, 2019 at 5:53
  • 1
    $\begingroup$ I added Jesper's clarifying comments to the answer. $\endgroup$ Oct 16, 2019 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.