Regression residual distribution assumptions

Question

Why is it necessary to place the distributional assumption on the errors, i.e.

$y_i = X\beta + \epsilon_{i}$, with $\epsilon_{i} \sim \mathcal{N}(0,\sigma^{2})$.

Why not write

$y_i = X\beta + \epsilon_{i}$, with $y_i \sim \mathcal{N}(X\hat{\beta},\sigma^{2})$,

where in either case $\epsilon_i = y_i - \hat{y}$.
I've seen it stressed that the distributional assumptions are placed on the errors, not the data, but without explanation.

I'm not really understanding the difference between these two formulations. Some places I see distributional assumptions being placed on the data (Bayesian lit. it seems mostly), but most times the assumptions are placed on the errors.

When modelling, why would/should one choose to begin with assumptions on one or the other?

Answer 1

In a linear regression setting it is common to do analysis and derive results conditional on $X$, i.e. conditional on "the data". Thus, what you need is that $y\mid X $ is normal, that is, you need $\epsilon$ to be normal. As Peter Flom's example illustrates, one can have normality of $\epsilon$ without having normality of $y$, and, thus, since what you need is normality of $\epsilon$, that's the sensible assumption.

Answer 2

I would write the second definition as

$y_i \sim \mathcal{N}(X_i\beta, \sigma^2)$

or (as Karl Oskar suggests +1)

$y_i|X_i \sim \mathcal{N}(X_i\beta, \sigma^2)$

i.e. the modelling assumption is that the response variable is normally distributed around the regression line (which is an estimate of the conditional mean), with constant variance $\sigma^2$. This is not the same thing as suggesting that $y_i$ are normally distributed, because the mean of the distribution depends on $X_i$.

I think I have seen similar formulations to this in the machine learning literature; as far as I can see it is equivalent to the first definition, all I have done is to rexpress the second formulation a little differently to eliminate the $\epsilon_i$'s and the $\hat{y}$'s.

Answer 3

The difference is easiest to illustrate with an example. Here's a simple one:

Suppose Y is bimodal, with the modality accounted for by an independent variable. E.g. suppose Y is height and your sample (for whatever reason) consists of jockeys and basketball players. e.g. in R

set.seed(123)
tall <- rnorm(100, 78, 3)
short <- rnorm(100, 60, 3)

height <- c(tall, short)
sport <- c(rep("B", 100), rep("H",100))

plot(density(height))

m1 <- lm(height~sport)
plot(m1)

the first density is very non-normal. But the residuals from the model are extremely close to normal.

As to why the restrictions are placed this way - I will let someone else answer that one.

Answer 4

You need to add a suscripted i to your second formulation:
$$ y_i\sim\mathcal N(\hat y_i,\sigma^2_\varepsilon) $$ because $\hat y$ needs to be able to vary along with $\bf x_i$.

---

That having been noted, what is $\hat y_i$? It is $\bf x_i\boldsymbol{\hat\beta}$. This leads to the formulation @DikranMarsupial presents:
$$ y_i\sim\mathcal N({\bf x_i}\boldsymbol{\hat\beta},\sigma^2_\varepsilon) $$ It is worth recognizing that this is exactly the same as your first formulation, because both stipulate normal distributions and the expected values are equal. That is:
\begin{align} E[{\bf x_i}\boldsymbol{\hat\beta}] &= E[{\bf x_i}\boldsymbol{\hat\beta} + E[\mathcal N(0, \sigma^2_\varepsilon)]] \\ &= E[{\bf x_i}\boldsymbol{\hat\beta} + 0] \\ &= E[{\bf x_i}\boldsymbol{\hat\beta}] \end{align} (And obviously the variances are equal.) In other words, this is not a difference in assumptions, but simply a notational difference.

So the question becomes, is there a reason to prefer presenting the idea using the first formulation?

I think the answer is yes for two reasons:

1. People often confuse whether the raw data should be normally distributed (i.e., $Y$), or if the data conditional on $\bf X$ / the errors should be normally distributed (i.e., $Y|\bf X$ / $\varepsilon$), for example, see: What if residuals are normally distributed, but y is not?
2. People also often confuse what is supposed to be independent, the raw data or the errors. Moreover, we often mention the fact that something should be iid (independent and identically distributed); if you are thinking in terms of $Y|\bf X$ this can be another potential source of confusion, as $Y|\bf X$ can be independent, but cannot be identically distributed unless the null hypothesis holds (because the mean would vary).

I believe these confustions are more likely using the second formulation than the first.