# Expression for $\hat{\beta}$ in simple linear regression

For simple linear regression, I have in my notes that

$\hat{\beta} = \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{\sum(x_i-\bar{x})^2}=\frac{\sum(x_i-\bar{x})y_i}{\sum(x_i-\bar{x})^2}$

(was intended as a step in the proof that the covariance of the mean response and beta hat is 0)

It seems however that this is not true unless $\bar{y}=0$. Does this have something to do with the fact that we will take the covariance?

Or, have I just mis-copied, and it should be

$\hat{\beta} = \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{\sum(x_i-\bar{x})^2}=\sum\frac{(x_i-\bar{x})y_i}{\sum(x_i-\bar{x})^2}$,

as written here: Covariance term in simple linear regression?

From comments by @jbowman, @whuber

$\sum(x_i-\bar{x})\bar{y}=\sum x_i\bar{y}-\sum\bar{x}\bar{y}\implies\sum x_i\bar{y}=\sum\bar{x}\bar{y} \implies \bar{y}\sum x_i=\bar{x}\bar{y}\sum1 \implies \bar{y}\sum x_i=\bar{x}\bar{y}N\implies \bar{y}\bar{x}=\bar{x}\bar{y}$

• It is true regardless of $\bar{y}$ as long as you have a constant term in the regression as well. Apr 28 '18 at 16:40
• The algebraic law you are implicitly using is that $ab=0$ implies $a=0$ or $b=0$. You have applied it to the case $b=\bar y$ and $a=\sum(x_i-\bar x),$ but you have quoted only half the conclusion: namely, that $\bar y=0.$ Look at the other half.
– whuber
Apr 28 '18 at 18:50
• Given the unnecessarily length and complexity of the existing answers, it seems worthwhile pointing out that $$\sum((x_i-\bar x)\bar y) = \bar y \left(\sum(x_i-\bar x)\right) = \bar y(0) = 0.$$That's all there is to this question. @jbowman has provided the justification for why the sum of the $x$ residuals is zero.
– whuber
Apr 29 '18 at 13:08

## 2 Answers

Algebraically, it is: $$\hat{\beta} = \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{\sum(x_i-\bar{x})^2}=\frac{\sum(x_i-\bar{x})y_i-\sum(x_i-\bar{x})\bar{y}}{\sum(x_i-\bar{x})^2}=\\ \frac{\sum(x_i-\bar{x})y_i-\bar{y}\sum(x_i-\bar{x})}{\sum(x_i-\bar{x})^2}= \frac{\sum(x_i-\bar{x})y_i-\bar{y}\cdot 0}{\sum(x_i-\bar{x})^2}=\\ \frac{\sum(x_i-\bar{x})y_i}{\sum(x_i-\bar{x})^2}=\sum\frac{(x_i-\bar{x})y_i}{\sum(x_j-\bar{x})^2}.$$ Note: $$\frac{\sum a_i}{\sum b_i}=\frac{\sum a_i}{\sum b_j}=\frac{a_1+a_2+\cdots+a_n}{\sum b_j}=\frac{a_1}{\sum b_j}+\frac{a_2}{\sum b_j}+\cdots+\frac{a_n}{\sum b_j}=\sum \frac{a_i}{\sum b_j}.$$

• in the second line the right hand side is not correct because you don't normalize! But you could fix this if you change the loss function by a factor of $\frac{1}{N}$ Apr 29 '18 at 8:32

So first, we want to regress on $y$ and assume, so that

$\hat{y} = \beta (x-\bar{x})$ (in vectorial notation) with the loss function

$L = \sum_{i} (y_{i}-\hat{y}_{i})^{2} = \sum_{i} (y_{i}-\beta (x_{i}-\bar{x}))^{2}$

As it is already said, you can either regress on $y$ or on $y-\bar{y}$. The difference just lies in an additional constant so this will only change $\beta_{0}$ (the y-axis section).

Now we want to find the optimal beta, therefore

$\frac{\partial}{\partial \beta} L = \sum_{i} 2(y_{i}-\hat{\beta} (x_{i}-\bar{x}))\cdot (x_{i}-\bar{x}) = 0$

We can drop the factor of $2$ and rearrange the equation to

$\sum_{i} y (x_{i}-\bar{x})=\sum_{i} \hat{\beta} (x_{i}-\bar{x})^{2}$ $\Rightarrow \hat{\beta} = \frac{\sum_{i}y_{i}(x_{i}-\bar{x})}{\sum_{i}(x_{i}-\bar{x})^{2}}$

This last term on the right handside can be rewritten in the following ways $\frac{\sum_{i}y_{i}( x_{i}-\bar{x})}{\sum_{i}(x_{i}-\bar{x})^{2}} = \sum_{i}\frac{y_{i}(x_{i}-\bar{x})}{\sum_{i}(x_{i}-\bar{x})^{2}}$

Just be aware that the denominator is summed first.