2
$\begingroup$

Let us say we have a linear model $$ Y=\beta_0+\beta_1X+\varepsilon, \quad \varepsilon|X\sim i.i.N(0,\sigma^2). $$ Suppose estimation yields $(\hat\beta_0,\hat\beta_1,\hat\sigma^2)=(5,2,9)$.

Question 1: Shall one write $$ Y=5+2X+\hat\varepsilon $$ or $$ Y=5+2X+\varepsilon? $$ I believe the former makes more sense, since the coefficient estimates $(5,2)$ need not perfectly coincide with the true coefficients $(\beta_0,\beta_1)$ and so $\hat\varepsilon$ will not perfectly coincide with $\varepsilon$. On the other hand, I do not think I commonly see $\hat\varepsilon$ in such situations...

Question 2: What can be said about the estimated counterpart of $\varepsilon|X\sim i.i.N(0,\sigma^2)$? I think that $\hat\varepsilon$s are dependent, so one $i.$ from $i.i.N$ drops. Not sure whether conditioning on $X$ makes sense and if it does, what role it plays. The distribution has an estimated imposed mean of $0$ and variance of $9$, but other than that, can we say anything more? What is the most we can say that would be correct?

$\endgroup$
0
2
$\begingroup$

Q1: It is true that $y_i = \hat\beta_0 + \hat\beta_1 x_i + \hat\epsilon_i$ for each observation $i$, largely because this is the definition the estimated residual $\hat\epsilon_i$. Note the estimated residual is not a parameter which is estimated--instead it is constructed. On the other hand, the latter expression assumes you've exactly found the error $\epsilon$, which is of course not true.

Q2: In the case that $y \sim \mathcal{N}(X\beta, \sigma^2 I)$, which is a default linear model, the estimated residual $\hat{\epsilon} \sim \mathcal{N} \left(0, \sigma^2 \left( I - H \right) \right)$, where $H=X(X^TX)^{-1}X^T$ is the hat matrix. This fully characterizes the distribution of the estimated residuals.

$\endgroup$
7
  • $\begingroup$ That is very helpful. Some follow ups: 1. Does it make sense to use $\hat\varepsilon|X$ instead of $\hat\varepsilon$? 2. Does the $i.i.$ drop only because the residuals are dependent or are there more reasons for that? 3. What if the normality assumption is violated? Since we rarely can be sure it holds, I was hesitant to use $N$ for $\hat\varepsilon$. While I did not mind that for $\varepsilon$ since there it serves as an explicit assumption and nothing more, I am concerned about using it for $\hat\varepsilon$ it would serve as a statement of fact (a finding). $\endgroup$ – Richard Hardy May 28 at 16:44
  • $\begingroup$ @RichardHardy 1. Yes, that's fine to use. Most people treat the covariates as fixed, so it's mostly a matter of preference. 2. my response to Q2 is describing a vector, not each entry. 3. In that case, the distribution of the residuals is just a linear transformation of the distribution of y, whatever it is. So its $(I-H) \times (\text{distribution of y})$. $\endgroup$ – user257566 May 28 at 19:20
  • $\begingroup$ @RichardHardy does that make sense? $\endgroup$ – user257566 Jun 6 at 22:37
  • $\begingroup$ Yes, I think so. How does your vector notation deal with i.i.d. though? It is an assumption I would not like to skip, I would like it to be explicit. $\endgroup$ – Richard Hardy Jun 7 at 6:27
  • $\begingroup$ @RichardHardy It automatically deals with that. To help explain vector notation, let's briefly talk about the normal distribution, which may be more concrete. You could have $y \sim \mathcal{N}(X \beta, \sigma^2 I)$, in which case the entries of $y$ are independent and have constant variance. You could also have $y \sim \mathcal{N}(X \beta, \Sigma)$, where $\Sigma$ is an arbitrary positive semi definite matrix; in this case, the covariance between the entries of $y$ is whatever you specify. $\endgroup$ – user257566 Jun 7 at 14:49
2
$\begingroup$

Q1

A simple way to circumvent this is to write instead:

$$\hat{y} = 5 + 2x$$

This avoids any confusion over $\epsilon \neq \text{residuals}$. The next part is a matter of opinion, but I also think it looks better, because $\hat{\epsilon}$ implies the error is estimated, while it is $\boldsymbol{\beta}$ that is estimated and the residual is just a remainder of that estimating procedure.

Q2

The assumptions of ordinary linear regression include that the error is independently, identically distributed and independent of $X$.$^{[1]}$ Conditioning on $X$ is therefore not necessary and you can simply write $\epsilon$ in place of $\epsilon|X$.

In a textbook, the $i.i.$ part might be elucidating, but for a paper, using normal linear regression implies the errors are independent. If they weren't, any standard errors and $p$-values generated by the model wouldn't be correct anyway.

Lastly, a small point, but the error mean is assumed to be zero and the residuals will by definition sum to $0$ when using ordinary least squares, so I wouldn't say that the mean was estimated to be $0$.


$[1]$: See e.g. Fox, John. Applied regression analysis and generalized linear models. Los Angeles: Sage, 2008

$\endgroup$
3
  • $\begingroup$ Excellent minor point. I would like to press a bit harder on Q2: is $\hat\sigma^2=9$ the most we can say as the counterpart of the entire $\varepsilon|X\sim i.i.N(0,\sigma^2)$ part? $\endgroup$ – Richard Hardy May 28 at 16:34
  • $\begingroup$ Also, regarding the first paragraph under Q2, my notation makes the assumptions you have listed explicit. I wonder if concealing one of the assumptions by skipping $|X$ is justified. If it is, why not conceal any other assumption by omitting some other part in the notation (e.g. omitting $i.i.$)? I think it would be most consistent to keep the entire thing. $\endgroup$ – Richard Hardy May 28 at 16:42
  • $\begingroup$ Ping........... $\endgroup$ – Richard Hardy Jun 7 at 6:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.