# Regression residual distribution assumptions

Why is it necessary to place the distributional assumption on the errors, i.e.

$y_i = X\beta + \epsilon_{i}$, with $\epsilon_{i} \sim \mathcal{N}(0,\sigma^{2})$.

Why not write

$y_i = X\beta + \epsilon_{i}$, with $y_i \sim \mathcal{N}(X\hat{\beta},\sigma^{2})$,

where in either case $\epsilon_i = y_i - \hat{y}$.
I've seen it stressed that the distributional assumptions are placed on the errors, not the data, but without explanation.

I'm not really understanding the difference between these two formulations. Some places I see distributional assumptions being placed on the data (Bayesian lit. it seems mostly), but most times the assumptions are placed on the errors.

When modelling, why would/should one choose to begin with assumptions on one or the other?

• First, it's not "necessary", it depends what you intend to do. There are some good answers, but I think the crux is the underlying assumption of causality, in the sense of the Xs "causing" the y, and if you look at it that way you see that the distribution of y is "caused" by the distribution of the rhs, which is to say the Xs and the errors (if any). You can do plenty of econometrics with very limited distributional assumptions and, in particular, without normality. Thank God. – PatrickT Jan 28 '14 at 20:54
• $\hat y$ is not $X\beta$, and the population mean of the $y$'s isn't the same as the sample estimate of it. Which is to say that the second thing is not actually the same thing as the first, but if you replace it with its expectation ($E(\hat y) = E(y) = X\beta$), the two would be equivalent. – Glen_b Jan 28 '14 at 21:36
• What is $\hat{y}$? And if $y_i$ varies with $i$, why doesn't $X\beta$ vary? Please make up your mind which notation you want to use, the vector or matrix. Now if we assume that $\hat{y}=X\hat\beta$ your notation is more than bizzare: $y_i\sim N(x_i'(\sum x_jx_j')^{-1}\sum x_jy_j,\sigma^2)$, i.e. you define distribution of $y_i$ in terms of itself and all the other observations $y_j$! – mpiktas Jan 29 '14 at 8:26
• I've downvoted the question because I think the notation is confusing and this already resulted in several subtly conflicting answers. – mpiktas Jan 29 '14 at 8:31

In a linear regression setting it is common to do analysis and derive results conditional on $X$, i.e. conditional on "the data". Thus, what you need is that $y\mid X$ is normal, that is, you need $\epsilon$ to be normal. As Peter Flom's example illustrates, one can have normality of $\epsilon$ without having normality of $y$, and, thus, since what you need is normality of $\epsilon$, that's the sensible assumption.

I would write the second definition as

$y_i \sim \mathcal{N}(X_i\beta, \sigma^2)$

or (as Karl Oskar suggests +1)

$y_i|X_i \sim \mathcal{N}(X_i\beta, \sigma^2)$

i.e. the modelling assumption is that the response variable is normally distributed around the regression line (which is an estimate of the conditional mean), with constant variance $\sigma^2$. This is not the same thing as suggesting that $y_i$ are normally distributed, because the mean of the distribution depends on $X_i$.

I think I have seen similar formulations to this in the machine learning literature; as far as I can see it is equivalent to the first definition, all I have done is to rexpress the second formulation a little differently to eliminate the $\epsilon_i$'s and the $\hat{y}$'s.

The difference is easiest to illustrate with an example. Here's a simple one:

Suppose Y is bimodal, with the modality accounted for by an independent variable. E.g. suppose Y is height and your sample (for whatever reason) consists of jockeys and basketball players. e.g. in R

set.seed(123)
tall <- rnorm(100, 78, 3)
short <- rnorm(100, 60, 3)

height <- c(tall, short)
sport <- c(rep("B", 100), rep("H",100))

plot(density(height))

m1 <- lm(height~sport)
plot(m1)


the first density is very non-normal. But the residuals from the model are extremely close to normal.

As to why the restrictions are placed this way - I will let someone else answer that one.

• Thank you! I see what you mean with a bimodal distribution. Follow up question: What if the variances of the data are different, (heteroscedasticity?) Say.. all jockeys are small right, but the heights of basketball players ranges a lot. Maybe for them, tall <- rnorm(100,78,10). How does a situation like this change your assumptions on either $y_i$ or $\epsilon_i$? – bill_e Jan 28 '14 at 11:36
• In that case, heteroscedasticity would be a problem and you would need to use some other form of regression, or possibly some transformation, or you could add another variable (in this silly example, position played in basketball might do it). – Peter Flom Jan 28 '14 at 13:14
• I'm not sure the formulation is intended to suggest that the ys are normally distributed, just that they have a normal conditional distribution. – Dikran Marsupial Jan 28 '14 at 16:15

$$y_i\sim\mathcal N(\hat y_i,\sigma^2_\varepsilon)$$ because $\hat y$ needs to be able to vary along with $\bf x_i$.

That having been noted, what is $\hat y_i$? It is $\bf x_i\boldsymbol{\hat\beta}$. This leads to the formulation @DikranMarsupial presents:
$$y_i\sim\mathcal N({\bf x_i}\boldsymbol{\hat\beta},\sigma^2_\varepsilon)$$ It is worth recognizing that this is exactly the same as your first formulation, because both stipulate normal distributions and the expected values are equal. That is:
\begin{align} E[{\bf x_i}\boldsymbol{\hat\beta}] &= E[{\bf x_i}\boldsymbol{\hat\beta} + E[\mathcal N(0, \sigma^2_\varepsilon)]] \\ &= E[{\bf x_i}\boldsymbol{\hat\beta} + 0] \\ &= E[{\bf x_i}\boldsymbol{\hat\beta}] \end{align} (And obviously the variances are equal.) In other words, this is not a difference in assumptions, but simply a notational difference.

So the question becomes, is there a reason to prefer presenting the idea using the first formulation?

I think the answer is yes for two reasons:

1. People often confuse whether the raw data should be normally distributed (i.e., $Y$), or if the data conditional on $\bf X$ / the errors should be normally distributed (i.e., $Y|\bf X$ / $\varepsilon$), for example, see: What if residuals are normally distributed, but y is not?
2. People also often confuse what is supposed to be independent, the raw data or the errors. Moreover, we often mention the fact that something should be iid (independent and identically distributed); if you are thinking in terms of $Y|\bf X$ this can be another potential source of confusion, as $Y|\bf X$ can be independent, but cannot be identically distributed unless the null hypothesis holds (because the mean would vary).

I believe these confustions are more likely using the second formulation than the first.

• @Glen_b, I don't follow your comment. My claim is not that $\hat y$ is equal to $X\beta$, but rather that $\hat y_i$ is equal to $\bf x_i\boldsymbol{\hat\beta}$. The subscripted $_i$ indexing the observations is relevant. The idea is that the predicted value, $\hat y_i$, for a given observation is $\bf x_i\boldsymbol{\hat\beta}$. This has nothing to do w/ the population mean of $Y$. (It appears that I had forgotten to add hats to my betas, though; I've corrected that now.) – gung - Reinstate Monica Jan 28 '14 at 22:38
• @Glen_b if it were the sample mean it would be $\bar{y}$ rather than $\hat{y}$. I initially found the notation confusing as well, but the fact that $\hat{y} = X\beta$ follows from the statements that $y_i = X\beta + \epsilon_i$ and $\epsilon_i = y_i - \hat{y}$. For both of these things to be true, $\hat{y}$ can only be $X\beta$. – Dikran Marsupial Jan 29 '14 at 8:29