I think in common literature about statstics the authors are often very imprecise when it comes to residuals and errors. So far, I could not work that difference out completely and therefore have several questions.
Setting: Let us consider the simple linear regression model, $$ Y = \beta_0 + \beta_1 x + \epsilon,$$ or for a specific value $Y=y_i$ and $x=x_i$,
$$ y_i = \beta_0 + \beta_1 x_i + \epsilon_i$$ with the error terms being normally distributed, i.e., $\epsilon_i$ ~ $N(0, \sigma^2)$. In this setting we consider the $Y=y_i$ and the errors $\epsilon_i$ to be random variables, and the independent variables $x_i$ to be nonrandom variables. The parameters $\beta_0$ and $\beta_1$ are unknown and fixed.
Therefore we have $E[Y] = \beta_0 + \beta_1 x$ and $Var[Y] = \sigma^2$, which means the $Y=y_i$ are also normally distributed according to $Y$~$N(\beta_0 + \beta_1 x_i, \sigma^2)$.
The errors $\epsilon_i$ are now defined as the deviations of the observations $y_i$ from the "true" (deterministic model) $E[Y] = \beta_0 + \beta_1 x$, i.e. $$ y_i - E[Y=y_i] = (\beta_0 + \beta_1 x_i + \epsilon_i)-(\beta_0 + \beta_1 x_i) = \epsilon_i.$$
We now want to estimate the model $E[Y] = \beta_0 + \beta_1 x$, as the coefficients $\beta_0$ and $\beta_1$ are unkown. We do so by $$ \hat{Y} = \hat{\beta_0} + \hat{\beta_1} x.$$
The residuals, lets call them $\delta_i$, are definded (as I understand it) as the deviations of the observations $y_i$ from the estimated model $\hat{y_i}$, i.e. $$\delta_i = y_i - \hat{y_i}.$$
Now the questions:
1.) In least squares estimation some authors reduce the squared sum of errors (SSE), $\sum \epsilon_i^2$, and some reduce the residual sum of square (RSS), $\sum \delta_i^2$, which is obviously not the same thing. Some even write they reduce $\sum \left(y_i - \hat{y_i}\right)^2$ but then they reduce the SSE, which is not even consistent within their own chosen framework. What is now the correct procedure of least square estimation, reducing SSE or RSS?
2.) How comes that there are so many different definitions of residuals, errors and least squares in various text books and wikipedia pages? Are the defintions of errors and residuals indeed not strictly clear, i.e. is it debatable what we consider to be the residuals and what the errors?
3.) Can we say anything about the distribution of the residuals, i.e. are they also normally distributed and if so with which mean and variance? It appears to me that for some reason it is generally assumed that the residuals are normal distributed, because it is often suggested to plot the residuals against the observation $y_i$ and check if they really are normal random distributed (which is expected to be the correct result).
4.) Some authors also write the estimated fromula as $ \hat{y} = \hat{\beta_0} + \hat{\beta_1} x_i + \hat{\epsilon_i}$, where they depict the residuals $\delta_i$ as $\hat{\epsilon_i}$. But I think this is a wrong formula because then $y_i - \hat{y_i}$, which is the defintion of the residuals, leads to a result that is different from the residuals. Also, as I understand it, the estimated model should not contain any random noise term as we want to plot a straight line. Is there any justification why some would write the residuals into the estimated formula?
EDIT: The books I was mainly looking at were:
i) Mathematical Statistics, Wackerly et al.: Here they reduce $\sum (y_i - \hat{y_i})^2$, but call this the sum of squared errors, i.e. they explicitly call $y_i - \hat{y_i}$ the errors. (p.569)
ii) An Introduction to Statistical Learning, James et al.: They reduce the residual sum of square and indeed define the residuals as I did above, i.e. $\delta_i = y_i - \hat{y_i}.$
iii) The Elements of Statistical Learning, Hastie et al.: They say that say that they would be minimizing the residual sum of squares (RSS) but what they actuall do is reducing the sum of squared errors, i.e. $\sum (y_i - \beta_0 + \beta_1 x_i)^2$, p.63
iv) Regression-Models, Methods and Applications, Fahrmeir et al.: They define the residuals as I did, i.e. as $\delta_i = y_i - \hat{y_i}$ and also discuss both residuals and errors but end up minimizing the sum of squared errors (SSE) in the least squares framework.