Linear Regression Understanding Least Squares

I'm reading the "Introduction to Statistical Learning" book and I have some questions regarding the book. The first one is the derivation of the least squares equation where the author talks about the residual, which effectively is the difference between the $$i$$-th observed response and the $$i$$-th response value that is predicted.

$$\epsilon_i = y_i - \hat{y}_i$$

I'm referring to page 62 of the book.

Later on in the same page the author derives the values for $$\beta_0$$ and $$\beta_1$$ and he says that it is done using some simple calculus. I would like to have or rather know the proof for it! Any suggestions?

• @Karolis: How did you get those math notations correct? Thanks for the edit! – sparkr Oct 15 '17 at 8:50
• you can press 'edit' and you should be able to see the raw text. It uses mathjax syntax I think but not sure. You put the text between \$\$ separators and follow a simple syntax. – Karolis Koncevičius Oct 15 '17 at 8:54
• Why is the "author derives values for $\beta_0$ and $\beta_1$" not a proof? – Sextus Empiricus Oct 15 '17 at 8:57
• mathworld.wolfram.com/LeastSquaresFitting.html inOLS, linear model coefficients are derived from a system of equations and can be got after matrix algebraic operations. – Alexey Burnakov Oct 15 '17 at 9:19
• For the mathematics, see math.meta.stackexchange.com/questions/5020/… -- if you're familiar with LaTeX, mathjax uses a subset of that. There are a number of derivations on site, so a search should turn up what you seek. – Glen_b -Reinstate Monica Oct 15 '17 at 11:44

I agree with you totally. I was also surprised by the lack of calculus for $$\beta_0$$ and $$\beta_1$$ in that book. Fortunately, it is quite straightforward to understand.

First of all, let's define residual - exactly as you wrote, it is the difference between $$i$$-th observed and $$i$$-th predicted value, however the equation is slightly different ($$i$$-th predicted value should be denoted by $$\hat y_i$$): $$\epsilon_i = y_i - \hat y_i$$

Having that, we can define sum of squared residuals (RSS) which is basically: $$RSS = \sum_{i=1}^N \epsilon_i^2 = \sum_{i=1}^N (y_i - \hat y_i)^2 = \sum_{i=1}^N (y_i - \hat \beta_0 - \hat\beta_1 x_i)^2$$ where $$N$$ is a number of observations.

Then, we can define our optimization problem of linear regression as: $$\min_{\hat\beta_0, \hat\beta_1} \sum_{i=1}^N (y_i - \hat\beta_0 - \hat\beta_1 x_i)^2$$

In order to solve above optimization problem with respect to regression parameters, we need to take the partial derivatives:

$$\frac{\partial RSS}{\partial \hat\beta_0} = \sum_{i=1}^N -2 (y_i - \hat\beta_0 - \hat\beta_1 x_i) = 0$$

$$\frac{\partial RSS}{\partial \hat\beta_1} = \sum_{i=1}^N -2 x_i (y_i - \hat\beta_0 - \hat\beta_1 x_i) = 0$$

Using above formulas, we can derive equations for both parameters.

To derive $$\beta_0$$, we are going to use $$\frac{\partial RSS}{\partial \hat\beta_0}$$. We may do that by get rid of constant $$-2$$ and use the fact that $$\sum_{i=1}^N x_i = N\bar x$$:

$$\sum_{i=1}^N y_i - \hat\beta_0 - \hat\beta_1 x_i = 0\\ N\bar y - N \hat\beta_0 - N \hat\beta_1 \bar x = 0\\ N \hat\beta_0 = N\bar y - N \hat\beta_1 \bar x\\ \hat\beta_0 = \bar y - \hat\beta_1 \bar x$$

Then we can solve $$\hat\beta_1$$ using $$\frac{\partial RSS}{\partial \hat\beta_1}$$ and already calculated $$\hat\beta_0$$: $$\sum_{i=1}^N x_i y_i - \hat\beta_0 x_i - \hat\beta_1 x_i^2 = 0\\ \sum_{i=1}^N x_i y_i - (\bar y - \hat\beta_1 \bar x) x_i - \hat\beta_1 x_i^2 = 0\\ \sum_{i=1}^N x_i y_i - \bar y \sum_{i=1}^N x_i + \hat\beta_1 \bar x \sum_{i=1}^N x_i - \hat\beta_1 \sum_{i=1}^N x_i^2 = 0\\ \sum_{i=1}^N x_i y_i - N\bar x \bar y + N \hat\beta_1 \bar x^2 - \hat\beta_1 \sum_{i=1}^N x_i^2 = 0\\ \hat\beta_1 (\sum_{i=1}^N x_i^2 - N {\bar x}^2) = \sum_{i=1}^N x_i y_i - N\bar x \bar y\\ \hat\beta_1 = \frac{\sum_{i=1}^N x_i y_i - N\bar x \bar y}{\sum_{i=1}^N x_i^2 - N {\bar x}^2}\\ \hat\beta_1 = \frac{\sum_{i=1}^N (x_i - \bar x)(y_i - \bar y)}{\sum_{i=1}^N (x_i - \bar x)^2}$$

I hope it will be helpful for you.

• Thanks for the explanation. But from where did you get the -2 all of a sudden? – sparkr Oct 15 '17 at 17:42
• @sparkr I got that by applying derivatives rule that $x^2 = 2x$. Since $\beta_0$ and $\beta_1$ have minuses before you have to multiple the equation in brackets by $-2$. – Jakub Wasikowski Oct 15 '17 at 21:00