1
$\begingroup$

For simple linear regression, I have in my notes that

$\hat{\beta} = \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{\sum(x_i-\bar{x})^2}=\frac{\sum(x_i-\bar{x})y_i}{\sum(x_i-\bar{x})^2}$

(was intended as a step in the proof that the covariance of the mean response and beta hat is 0)

It seems however that this is not true unless $\bar{y}=0$. Does this have something to do with the fact that we will take the covariance?

Or, have I just mis-copied, and it should be

$\hat{\beta} = \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{\sum(x_i-\bar{x})^2}=\sum\frac{(x_i-\bar{x})y_i}{\sum(x_i-\bar{x})^2}$,

as written here: Covariance term in simple linear regression?

From comments by @jbowman, @whuber

$\sum(x_i-\bar{x})\bar{y}=\sum x_i\bar{y}-\sum\bar{x}\bar{y}\implies\sum x_i\bar{y}=\sum\bar{x}\bar{y} \implies \bar{y}\sum x_i=\bar{x}\bar{y}\sum1 \implies \bar{y}\sum x_i=\bar{x}\bar{y}N\implies \bar{y}\bar{x}=\bar{x}\bar{y}$

$\endgroup$
  • 1
    $\begingroup$ It is true regardless of $\bar{y}$ as long as you have a constant term in the regression as well. $\endgroup$ – jbowman Apr 28 '18 at 16:40
  • 1
    $\begingroup$ The algebraic law you are implicitly using is that $ab=0$ implies $a=0$ or $b=0$. You have applied it to the case $b=\bar y$ and $a=\sum(x_i-\bar x),$ but you have quoted only half the conclusion: namely, that $\bar y=0.$ Look at the other half. $\endgroup$ – whuber Apr 28 '18 at 18:50
  • 1
    $\begingroup$ Given the unnecessarily length and complexity of the existing answers, it seems worthwhile pointing out that $$\sum((x_i-\bar x)\bar y) = \bar y \left(\sum(x_i-\bar x)\right) = \bar y(0) = 0.$$That's all there is to this question. @jbowman has provided the justification for why the sum of the $x$ residuals is zero. $\endgroup$ – whuber Apr 29 '18 at 13:08
1
$\begingroup$

Algebraically, it is: $$\hat{\beta} = \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{\sum(x_i-\bar{x})^2}=\frac{\sum(x_i-\bar{x})y_i-\sum(x_i-\bar{x})\bar{y}}{\sum(x_i-\bar{x})^2}=\\ \frac{\sum(x_i-\bar{x})y_i-\bar{y}\sum(x_i-\bar{x})}{\sum(x_i-\bar{x})^2}= \frac{\sum(x_i-\bar{x})y_i-\bar{y}\cdot 0}{\sum(x_i-\bar{x})^2}=\\ \frac{\sum(x_i-\bar{x})y_i}{\sum(x_i-\bar{x})^2}=\sum\frac{(x_i-\bar{x})y_i}{\sum(x_j-\bar{x})^2}.$$ Note: $$\frac{\sum a_i}{\sum b_i}=\frac{\sum a_i}{\sum b_j}=\frac{a_1+a_2+\cdots+a_n}{\sum b_j}=\frac{a_1}{\sum b_j}+\frac{a_2}{\sum b_j}+\cdots+\frac{a_n}{\sum b_j}=\sum \frac{a_i}{\sum b_j}.$$

$\endgroup$
  • $\begingroup$ in the second line the right hand side is not correct because you don't normalize! But you could fix this if you change the loss function by a factor of $\frac{1}{N}$ $\endgroup$ – pythonic833 Apr 29 '18 at 8:32
0
$\begingroup$

So first, we want to regress on $y$ and assume, so that

$\hat{y} = \beta (x-\bar{x})$ (in vectorial notation) with the loss function

$L = \sum_{i} (y_{i}-\hat{y}_{i})^{2} = \sum_{i} (y_{i}-\beta (x_{i}-\bar{x}))^{2}$

As it is already said, you can either regress on $y$ or on $y-\bar{y}$. The difference just lies in an additional constant so this will only change $\beta_{0}$ (the y-axis section).

Now we want to find the optimal beta, therefore

$\frac{\partial}{\partial \beta} L = \sum_{i} 2(y_{i}-\hat{\beta} (x_{i}-\bar{x}))\cdot (x_{i}-\bar{x}) = 0$

We can drop the factor of $2$ and rearrange the equation to

$\sum_{i} y (x_{i}-\bar{x})=\sum_{i} \hat{\beta} (x_{i}-\bar{x})^{2} $ $\Rightarrow \hat{\beta} = \frac{\sum_{i}y_{i}(x_{i}-\bar{x})}{\sum_{i}(x_{i}-\bar{x})^{2}}$

This last term on the right handside can be rewritten in the following ways $ \frac{\sum_{i}y_{i}( x_{i}-\bar{x})}{\sum_{i}(x_{i}-\bar{x})^{2}} = \sum_{i}\frac{y_{i}(x_{i}-\bar{x})}{\sum_{i}(x_{i}-\bar{x})^{2}} $

Just be aware that the denominator is summed first.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.