# Estimated linear model: notation, properties of residuals

Let us say we have a linear model $$Y=\beta_0+\beta_1X+\varepsilon, \quad \varepsilon|X\sim i.i.N(0,\sigma^2).$$ Suppose estimation yields $$(\hat\beta_0,\hat\beta_1,\hat\sigma^2)=(5,2,9)$$.

Question 1: Shall one write $$Y=5+2X+\hat\varepsilon$$ or $$Y=5+2X+\varepsilon?$$ I believe the former makes more sense, since the coefficient estimates $$(5,2)$$ need not perfectly coincide with the true coefficients $$(\beta_0,\beta_1)$$ and so $$\hat\varepsilon$$ will not perfectly coincide with $$\varepsilon$$. On the other hand, I do not think I commonly see $$\hat\varepsilon$$ in such situations...

Question 2: What can be said about the estimated counterpart of $$\varepsilon|X\sim i.i.N(0,\sigma^2)$$? I think that $$\hat\varepsilon$$s are dependent, so one $$i.$$ from $$i.i.N$$ drops. Not sure whether conditioning on $$X$$ makes sense and if it does, what role it plays. The distribution has an estimated imposed mean of $$0$$ and variance of $$9$$, but other than that, can we say anything more? What is the most we can say that would be correct?

Q1: It is true that $$y_i = \hat\beta_0 + \hat\beta_1 x_i + \hat\epsilon_i$$ for each observation $$i$$, largely because this is the definition the estimated residual $$\hat\epsilon_i$$. Note the estimated residual is not a parameter which is estimated--instead it is constructed. On the other hand, the latter expression assumes you've exactly found the error $$\epsilon$$, which is of course not true.

Q2: In the case that $$y \sim \mathcal{N}(X\beta, \sigma^2 I)$$, which is a default linear model, the estimated residual $$\hat{\epsilon} \sim \mathcal{N} \left(0, \sigma^2 \left( I - H \right) \right)$$, where $$H=X(X^TX)^{-1}X^T$$ is the hat matrix. This fully characterizes the distribution of the estimated residuals.

• That is very helpful. Some follow ups: 1. Does it make sense to use $\hat\varepsilon|X$ instead of $\hat\varepsilon$? 2. Does the $i.i.$ drop only because the residuals are dependent or are there more reasons for that? 3. What if the normality assumption is violated? Since we rarely can be sure it holds, I was hesitant to use $N$ for $\hat\varepsilon$. While I did not mind that for $\varepsilon$ since there it serves as an explicit assumption and nothing more, I am concerned about using it for $\hat\varepsilon$ it would serve as a statement of fact (a finding). May 28, 2021 at 16:44
• @RichardHardy 1. Yes, that's fine to use. Most people treat the covariates as fixed, so it's mostly a matter of preference. 2. my response to Q2 is describing a vector, not each entry. 3. In that case, the distribution of the residuals is just a linear transformation of the distribution of y, whatever it is. So its $(I-H) \times (\text{distribution of y})$. May 28, 2021 at 19:20
• @RichardHardy does that make sense? Jun 6, 2021 at 22:37
• Yes, I think so. How does your vector notation deal with i.i.d. though? It is an assumption I would not like to skip, I would like it to be explicit. Jun 7, 2021 at 6:27
• @RichardHardy It automatically deals with that. To help explain vector notation, let's briefly talk about the normal distribution, which may be more concrete. You could have $y \sim \mathcal{N}(X \beta, \sigma^2 I)$, in which case the entries of $y$ are independent and have constant variance. You could also have $y \sim \mathcal{N}(X \beta, \Sigma)$, where $\Sigma$ is an arbitrary positive semi definite matrix; in this case, the covariance between the entries of $y$ is whatever you specify. Jun 7, 2021 at 14:49

### Q1

A simple way to circumvent this is to write instead:

$$\hat{y} = 5 + 2x$$

This avoids any confusion over $$\epsilon \neq \text{residuals}$$. The next part is a matter of opinion, but I also think it looks better, because $$\hat{\epsilon}$$ implies the error is estimated, while it is $$\boldsymbol{\beta}$$ that is estimated and the residual is just a remainder of that estimating procedure.

### Q2

The assumptions of ordinary linear regression include that the error is independently, identically distributed and independent of $$X$$.$$^{[1]}$$ Conditioning on $$X$$ is therefore not necessary and you can simply write $$\epsilon$$ in place of $$\epsilon|X$$.

In a textbook, the $$i.i.$$ part might be elucidating, but for a paper, using normal linear regression implies the errors are independent. If they weren't, any standard errors and $$p$$-values generated by the model wouldn't be correct anyway.

Lastly, a small point, but the error mean is assumed to be zero and the residuals will by definition sum to $$0$$ when using ordinary least squares, so I wouldn't say that the mean was estimated to be $$0$$.

$$[1]$$: See e.g. Fox, John. Applied regression analysis and generalized linear models. Los Angeles: Sage, 2008

• Excellent minor point. I would like to press a bit harder on Q2: is $\hat\sigma^2=9$ the most we can say as the counterpart of the entire $\varepsilon|X\sim i.i.N(0,\sigma^2)$ part? May 28, 2021 at 16:34
• Also, regarding the first paragraph under Q2, my notation makes the assumptions you have listed explicit. I wonder if concealing one of the assumptions by skipping $|X$ is justified. If it is, why not conceal any other assumption by omitting some other part in the notation (e.g. omitting $i.i.$)? I think it would be most consistent to keep the entire thing. May 28, 2021 at 16:42
• Ping........... Jun 7, 2021 at 6:27