Linear Regression Understanding Least Squares I'm reading the "Introduction to Statistical Learning" book and I have some questions regarding the book. The first one is the derivation of the least squares equation where the author talks about  the residual, which effectively is the difference between the $i$-th observed response and the $i$-th response value that is predicted.
$\epsilon_i = y_i - \hat{y}_i$
I'm referring to page 62 of the book.
Later on in the same page the author derives the values for $\beta_0$ and $\beta_1$ and he says that it is done using some simple calculus. I would like to have or rather know the proof for it! Any suggestions?
 A: I agree with you totally. I was also surprised by the lack of calculus for $\beta_0$ and $\beta_1$ in that book. Fortunately, it is quite straightforward to understand. 
First of all, let's define residual - exactly as you wrote, it is the difference between $i$-th observed and $i$-th predicted value, however the equation is slightly different ($i$-th predicted value should be denoted by $\hat y_i $):
$$ \epsilon_i = y_i - \hat y_i$$
Having that, we can define sum of squared residuals (RSS) which is basically:
$$ RSS = \sum_{i=1}^N \epsilon_i^2 = \sum_{i=1}^N (y_i - \hat y_i)^2 = \sum_{i=1}^N (y_i - \hat \beta_0  - \hat\beta_1 x_i)^2 $$
where $N$ is a number of observations.
Then, we can define our optimization problem of linear regression as:
$$\min_{\hat\beta_0, \hat\beta_1} \sum_{i=1}^N (y_i - \hat\beta_0 - \hat\beta_1 x_i)^2 $$
In order to solve above optimization problem with respect to regression parameters, we need to take the partial derivatives:
$$\frac{\partial RSS}{\partial \hat\beta_0} = \sum_{i=1}^N -2 (y_i - \hat\beta_0 - \hat\beta_1 x_i) = 0$$
$$\frac{\partial RSS}{\partial \hat\beta_1} = \sum_{i=1}^N -2 x_i (y_i - \hat\beta_0 - \hat\beta_1 x_i) = 0$$
Using above formulas, we can derive equations for both parameters.
To derive $\beta_0$, we are going to use $\frac{\partial RSS}{\partial \hat\beta_0}$. We may do that by get rid of constant $-2$ and use the fact that $\sum_{i=1}^N x_i = N\bar x$:
$$
\sum_{i=1}^N y_i - \hat\beta_0 - \hat\beta_1 x_i = 0\\
N\bar y - N \hat\beta_0 - N \hat\beta_1 \bar x = 0\\
N \hat\beta_0 = N\bar y - N \hat\beta_1 \bar x\\
\hat\beta_0 = \bar y - \hat\beta_1 \bar x
$$
Then we can solve $\hat\beta_1$ using $\frac{\partial RSS}{\partial \hat\beta_1}$ and already calculated $\hat\beta_0$:
$$
\sum_{i=1}^N x_i y_i - \hat\beta_0 x_i - \hat\beta_1 x_i^2 = 0\\
\sum_{i=1}^N x_i y_i - (\bar y - \hat\beta_1 \bar x) x_i - \hat\beta_1 x_i^2 = 0\\
\sum_{i=1}^N x_i y_i - \bar y \sum_{i=1}^N x_i + \hat\beta_1 \bar x \sum_{i=1}^N x_i - \hat\beta_1 \sum_{i=1}^N x_i^2 = 0\\
\sum_{i=1}^N x_i y_i - N\bar x \bar y + N \hat\beta_1 \bar x^2 - \hat\beta_1 \sum_{i=1}^N x_i^2 = 0\\
\hat\beta_1 (\sum_{i=1}^N x_i^2 - N {\bar x}^2) = \sum_{i=1}^N x_i y_i - N\bar x \bar y\\
\hat\beta_1 = \frac{\sum_{i=1}^N x_i y_i - N\bar x \bar y}{\sum_{i=1}^N x_i^2 - N {\bar x}^2}\\
\hat\beta_1 = \frac{\sum_{i=1}^N (x_i - \bar x)(y_i - \bar y)}{\sum_{i=1}^N (x_i - \bar x)^2}
$$
I hope it will be helpful for you.
