3
$\begingroup$

So we're looking for the coefficients $\beta_0, \beta_1$ such that we minimize $\sum_{i=1}^n\epsilon_i^2$ in $$Y_i = \beta_0 + \beta_1X_i + \epsilon_i$$ Meaning, $\beta_0$ and $\beta_1$ such that $$\sum_{i=1}^n\epsilon_i^2 = \sum_{i=1}^n\big(Y_i - \beta_0 - \beta_1X_i \big)^2$$ is minimal. We do that by solving a system of equations that we get by equating the partial derivatives with zero: $$\frac{d}{d\beta_0}\sum_{i=1}^n\big(Y_i - \beta_0 - \beta_1X_i \big)^2 = 0$$ $$\frac{d}{d\beta_1}\sum_{i=1}^n\big(Y_i - \beta_0 - \beta_1X_i \big)^2 = 0$$ This gives us two equations: $$-2\sum_{i=1}^n\big(Y_i-\beta_1X_i - \beta_0\big) = 0$$ $$-2\sum_{i=1}^nX_i\big(Y_i-\beta_1X_i - \beta_0\big) = 0$$ which further reduce to $$\sum_{i=1}^nY_i - \beta_1\sum_{i=1}^nX_i - n\beta_0 = 0$$ $$\sum_{i=1}^nX_iY_i - \beta_1\sum_{i=1}^nX_i^2 - \beta_0\sum_{i=1}^nX_i = 0$$ And I have managed to solve this system myself and I got the following solution for $\beta_1$:

$$\beta_1 = \frac{\sum_{i=1}^nX_iY_i - n\bar{X_n}\bar{Y_n}}{\sum_{i=1}^nX_i^2 - n\bar{X_n}^2}$$ This solution matches with the solution given in my textbook. However, my textbook also adds one more equality for which it gives no explanation. So the textbook version of the final solution is this: $$\beta_1 = \frac{\sum_{i=1}^nX_iY_i - n\bar{X_n}\bar{Y_n}}{\sum_{i=1}^nX_i^2 - n\bar{X_n}^2} = \frac{\sum_{i=1}^n(X_i-\bar{X_n})(Y_i-\bar{Y_n})}{\sum_{i=1}^n(X_i-\bar{X_n})^2}$$ At first I thought, since they showed no work, it must be very simple to derive the RHS from the LHS. It may be. But I've tried it for a while now and I just can't seem to do it. Tried rewriting the sample means as averages and all kinds of stuff but I either get right back where I started or the expression gets too complicated so I have to start all over again.

Any ideas? Thanks.

$\endgroup$
8
  • $\begingroup$ I believe this question has appeared, in one guise or another, many dozens of times here on CV. Unfortunately it's almost impossible to search out answers because all the posts are in terms of the equations. Let me just point out, then, that the algebra is identical in the numerator and denominator and relies on the fact that the residuals $X_i-\bar X$ sum to zero. $\endgroup$
    – whuber
    Aug 27, 2020 at 14:24
  • $\begingroup$ If you derive it from the covariance, the formula might become a bit more obvious as well $\endgroup$
    – Firebug
    Aug 27, 2020 at 14:36
  • $\begingroup$ Thank you both for the answers. @whuber I've tried using that fact and unfortunately it still gets me nowhere. Firebug, The formula is obvious when thinking about covariance, however here I'm just looking for a way to derive it from the other one. $\endgroup$
    – Koy
    Aug 27, 2020 at 14:45
  • $\begingroup$ @Koy I added an answer, hoping it helps you get there, but if you need any more help just say it and we'll sort it out :) $\endgroup$
    – Firebug
    Aug 27, 2020 at 14:52
  • $\begingroup$ The easiest way I know of seeing this is to recognize that both the numerator and denominator remain unchanged when you add a constant value to all the $X_i$ or to all the $Y_i$. This permits you to choose a suitable constant that makes $\bar X =\bar Y = 0$ and you're done because there's nothing left to show. $\endgroup$
    – whuber
    Aug 27, 2020 at 14:54

1 Answer 1

1
$\begingroup$

Let's start from the LHS expression. I rearranged it, removing the $n$ by reintroducing the averages into the summations:

$$ \beta_1 = \frac{ \sum_{i=1}^n (X_iY_i - \bar{X_n}\bar{Y_n}) }{ \sum_{i=1}^n (X_i^2 - \bar{X_n}^2) }$$

In the denominator we can complete squares. In the numerator, we can add and subtract missing terms.

$$ \beta_1 = \frac{ \sum_{i=1}^n ((X_i - \bar{X_n})(Y_i - \bar{Y_n}) \color{red}{+ \bar{X_n}Y_i + X_i\bar{Y_n} - 2\bar{X_n}\bar{Y_n}}) }{ \sum_{i=1}^n ((X_i - \bar{X_n})^2 \color{red}{+ 2X_i\bar{X_n} - 2\bar{X_n}^2}) }$$

Can you now prove that the summations in red are both equal to zero?

$\endgroup$
2
  • 1
    $\begingroup$ Thanks, I actually managed to do it in the meantime. I guess the problem was that it's a lot easier to do it when you start from the RHS because all the terms are there so you don't need to add/subtract and complete squares. So I did it from the RHS to the LHS and then just reverse-engineered it. Thanks anyway. $\endgroup$
    – Koy
    Aug 27, 2020 at 14:54
  • 1
    $\begingroup$ @Koy I'm glad you made it. In exams or in classes, it's often allowed to 'reverse-engineer', so good that you deduced that as well. As we talked earlier, you can also see that the person that first derived the relation between the LHS and RHS expressions saw the connection with the covariance/variance ratio, and wanted to express it in more familiar formulas. $\endgroup$
    – Firebug
    Aug 27, 2020 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.