Errors and residuals in linear regression

Question

I think in common literature about statstics the authors are often very imprecise when it comes to residuals and errors. So far, I could not work that difference out completely and therefore have several questions.

Setting: Let us consider the simple linear regression model, $$ Y = \beta_0 + \beta_1 x + \epsilon,$$ or for a specific value $Y=y_i$ and $x=x_i$,

$$ y_i = \beta_0 + \beta_1 x_i + \epsilon_i$$ with the error terms being normally distributed, i.e., $\epsilon_i$ ~ $N(0, \sigma^2)$. In this setting we consider the $Y=y_i$ and the errors $\epsilon_i$ to be random variables, and the independent variables $x_i$ to be nonrandom variables. The parameters $\beta_0$ and $\beta_1$ are unknown and fixed.

Therefore we have $E[Y] = \beta_0 + \beta_1 x$ and $Var[Y] = \sigma^2$, which means the $Y=y_i$ are also normally distributed according to $Y$~$N(\beta_0 + \beta_1 x_i, \sigma^2)$.

The errors $\epsilon_i$ are now defined as the deviations of the observations $y_i$ from the "true" (deterministic model) $E[Y] = \beta_0 + \beta_1 x$, i.e. $$ y_i - E[Y=y_i] = (\beta_0 + \beta_1 x_i + \epsilon_i)-(\beta_0 + \beta_1 x_i) = \epsilon_i.$$

We now want to estimate the model $E[Y] = \beta_0 + \beta_1 x$, as the coefficients $\beta_0$ and $\beta_1$ are unkown. We do so by $$ \hat{Y} = \hat{\beta_0} + \hat{\beta_1} x.$$

The residuals, lets call them $\delta_i$, are definded (as I understand it) as the deviations of the observations $y_i$ from the estimated model $\hat{y_i}$, i.e. $$\delta_i = y_i - \hat{y_i}.$$

Now the questions:

1.) In least squares estimation some authors reduce the squared sum of errors (SSE), $\sum \epsilon_i^2$, and some reduce the residual sum of square (RSS), $\sum \delta_i^2$, which is obviously not the same thing. Some even write they reduce $\sum \left(y_i - \hat{y_i}\right)^2$ but then they reduce the SSE, which is not even consistent within their own chosen framework. What is now the correct procedure of least square estimation, reducing SSE or RSS?

2.) How comes that there are so many different definitions of residuals, errors and least squares in various text books and wikipedia pages? Are the defintions of errors and residuals indeed not strictly clear, i.e. is it debatable what we consider to be the residuals and what the errors?

3.) Can we say anything about the distribution of the residuals, i.e. are they also normally distributed and if so with which mean and variance? It appears to me that for some reason it is generally assumed that the residuals are normal distributed, because it is often suggested to plot the residuals against the observation $y_i$ and check if they really are normal random distributed (which is expected to be the correct result).

4.) Some authors also write the estimated fromula as $ \hat{y} = \hat{\beta_0} + \hat{\beta_1} x_i + \hat{\epsilon_i}$, where they depict the residuals $\delta_i$ as $\hat{\epsilon_i}$. But I think this is a wrong formula because then $y_i - \hat{y_i}$, which is the defintion of the residuals, leads to a result that is different from the residuals. Also, as I understand it, the estimated model should not contain any random noise term as we want to plot a straight line. Is there any justification why some would write the residuals into the estimated formula?

EDIT: The books I was mainly looking at were:

i) Mathematical Statistics, Wackerly et al.: Here they reduce $\sum (y_i - \hat{y_i})^2$, but call this the sum of squared errors, i.e. they explicitly call $y_i - \hat{y_i}$ the errors. (p.569)

ii) An Introduction to Statistical Learning, James et al.: They reduce the residual sum of square and indeed define the residuals as I did above, i.e. $\delta_i = y_i - \hat{y_i}.$

iii) The Elements of Statistical Learning, Hastie et al.: They say that say that they would be minimizing the residual sum of squares (RSS) but what they actuall do is reducing the sum of squared errors, i.e. $\sum (y_i - \beta_0 + \beta_1 x_i)^2$, p.63

iv) Regression-Models, Methods and Applications, Fahrmeir et al.: They define the residuals as I did, i.e. as $\delta_i = y_i - \hat{y_i}$ and also discuss both residuals and errors but end up minimizing the sum of squared errors (SSE) in the least squares framework.

Answer 1

I think there is a lot of confusion in this question (caused of course by authors that describe what they think linear regression means very bad/imprecise/up to wrong).

First of all we are given some data $(x_i, y_i)_{i=1,...,N}, x_i \in \mathbb{R}^d, y_i \in \mathbb{R}$ and we want to "make sense of it in form of a linear equation" (see below for a precise formulation). Now it may be the case that this model does absolutely not match the data, for example, if $d=1$ then it could be that $y_i = \sin(x_i)$ or so... Nevertheless one could use linear regression in order to write down a (shitty!) model for that but what you are looking for is the following version of linear regression:

Assume (some things about the data) then the linear regression model is the bestest model ever.

Now we are going to make this precise. First of all we assume that there is a probability space $\Omega$ and random variables $X_i : \Omega \to \mathbb{R}^d$ and $Y_i : \Omega \to \mathbb{R}$ and $\epsilon_i : \Omega \to \mathbb{R}$ and we assume that there are (as you call them) 'true' $\beta \in \mathbb{R}^d, b \in \mathbb{R}$ such that $$Y_i = \beta X_i + b + \epsilon_i$$ (as functions from $\Omega$ to $\mathbb{R}$) and we assume that there is a 'true' $\omega_0 \in \Omega$ such that $$x_i = X_i(\omega_0)$$ and $$y_i = Y_i(\omega_0)$$ and the $\epsilon_i$ are independent from the $X_i$ and the $\epsilon_i$ are iid. $\mathcal{N}(0, \sigma^2)$ distributed.

These assumptions mean that the data we are confronted with really comes from these random variables and they satisfy some relations. Then we can execute an algorithm in order to find approximations $\hat{\beta}, \hat{b}$ of $\beta, b$ such that when we are confronted with a new, unseen $x$, the equation $$\hat{y} = \hat{\beta} x + \hat{b}$$ will give the best (concerning some measurement, namely in average) approximation for the 'true' $y$ that belongs to that $x$.

1) Now we have to ask: What do these people mean when they write $\delta_i$, $\epsilon_i$, ...? They for sure do hardly know what the term 'random variable' really means from a mathematical point of view or they know it and jut ignore it, hence, they just use it for any symbol in their mind that is somewhat related to some kind of error. I guess that they mean $$\text{their}~ \delta_i = \hat{y}_i - y_i$$ i.e. given the current parameters $\hat{\beta}, \hat{b}$, what is the error to the $i$-th true training answer? This is a very concrete real number, not a random variable and this (well, the sum of the squares of them) is what you minimize in linear regression. When they write that they "minimize" something involving $\epsilon_i$ then we do not know what they mean: these are random variables that we cannot even change!!! How should this be minimized? Hence, I think that you are confused for the right reason: Whatever they write in the context of approximizing $\beta, b$, they almost always mean $\hat{y}_i - y_i$.

2) I have not seen such a book yet... I think it stems from the following: either the author does not know something about mathematics and precision (or does not give a sh*t about it) or he/she does not want to exhaust the audience with these (absolutely important) details... However, there are some questions in this direction here on se, see here or here and so forth... (shamelessly referring to questions and answers of myself here but probably you can find many more).

3) What do you mean by residiuals? Are you referring to the random variables $\epsilon_i$ or are you referring to $\hat{\beta}X_i + \hat{b} - Y_i$? I hardly doubt that the latter are normally distributed or so because this depends on the distribution on $X_i$ alone and these can have any distribution as long as they are in line with the corresponding $Y_i$!

4) Lack/ignorance of mathematical knowledge or they actually want to describe something else I guess... For example: One can analyze confidence intervals (i.e. we want to leave the perspective of one single line and for a fresh new unseen $x$ we want to give lower and upper bounds $y_l, y_u$ such that with abc% probability, $y_l \leq y \leq y_u$). Then uncertainty needs to come into play again.