Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Why is it necessary to place the distributional assumption on the errors, i.e.

$y_i = X\beta + \epsilon_{i}$, with $\epsilon_{i} \sim \mathcal{N}(0,\sigma^{2})$.

Why not write

$y_i = X\beta + \epsilon_{i}$, with $y_i \sim \mathcal{N}(X\hat{\beta},\sigma^{2})$,

where in either case $\epsilon_i = y_i - \hat{y}$.
I've seen it stressed that the distributional assumptions are placed on the errors, not the data, but without explanation.

I'm not really understanding the difference between these two formulations. Some places I see distributional assumptions being placed on the data (Bayesian lit. it seems mostly), but most times the assumptions are placed on the errors.

When modelling, why would/should one choose to begin with assumptions on one or the other?

share|improve this question
    
First, it's not "necessary", it depends what you intend to do. There are some good answers, but I think the crux is the underlying assumption of causality, in the sense of the Xs "causing" the y, and if you look at it that way you see that the distribution of y is "caused" by the distribution of the rhs, which is to say the Xs and the errors (if any). You can do plenty of econometrics with very limited distributional assumptions and, in particular, without normality. Thank God. –  PatrickT Jan 28 at 20:54
3  
$\hat y$ is not $X\beta$, and the population mean of the $y$'s isn't the same as the sample estimate of it. Which is to say that the second thing is not actually the same thing as the first, but if you replace it with its expectation ($E(\hat y) = E(y) = X\beta$), the two would be equivalent. –  Glen_b Jan 28 at 21:36
    
What is $\hat{y}$? And if $y_i$ varies with $i$, why doesn't $X\beta$ vary? Please make up your mind which notation you want to use, the vector or matrix. Now if we assume that $\hat{y}=X\hat\beta$ your notation is more than bizzare: $y_i\sim N(x_i'(\sum x_jx_j')^{-1}\sum x_jy_j,\sigma^2)$, i.e. you define distribution of $y_i$ in terms of itself and all the other observations $y_j$! –  mpiktas Jan 29 at 8:26
    
I've downvoted the question because I think the notation is confusing and this already resulted in several subtly conflicting answers. –  mpiktas Jan 29 at 8:31
add comment

4 Answers 4

up vote 6 down vote accepted

In a linear regression setting it is common to do analysis and derive results conditional on $X$, i.e. conditional on "the data". Thus, what you need is that $y\mid X $ is normal, that is, you need $\epsilon$ to be normal. As Peter Flom's example illustrates, one can have normality of $\epsilon$ without having normality of $y$, and, thus, since what you need is normality of $\epsilon$, that's the sensible assumption.

share|improve this answer
add comment

I would write the second definition as

$y_i \sim \mathcal{N}(X_i\beta, \sigma^2)$

or (as Karl Oskar suggests +1)

$y_i|X_i \sim \mathcal{N}(X_i\beta, \sigma^2)$

i.e. the modelling assumption is that the response variable is normally distributed around the regression line (which is an estimate of the conditional mean), with constant variance $\sigma^2$. This is not the same thing as suggesting that $y_i$ are normally distributed, because the mean of the distribution depends on $X_i$.

I think I have seen similar formulations to this in the machine learning literature; as far as I can see it is equivalent to the first definition, all I have done is to rexpress the second formulation a little differently to eliminate the $\epsilon_i$'s and the $\hat{y}$'s.

share|improve this answer
add comment

The difference is easiest to illustrate with an example. Here's a simple one:

Suppose Y is bimodal, with the modality accounted for by an independent variable. E.g. suppose Y is height and your sample (for whatever reason) consists of jockeys and basketball players. e.g. in R

set.seed(123)
tall <- rnorm(100, 78, 3)
short <- rnorm(100, 60, 3)

height <- c(tall, short)
sport <- c(rep("B", 100), rep("H",100))

plot(density(height))

m1 <- lm(height~sport)
plot(m1)

the first density is very non-normal. But the residuals from the model are extremely close to normal.

As to why the restrictions are placed this way - I will let someone else answer that one.

share|improve this answer
1  
Thank you! I see what you mean with a bimodal distribution. Follow up question: What if the variances of the data are different, (heteroscedasticity?) Say.. all jockeys are small right, but the heights of basketball players ranges a lot. Maybe for them, tall <- rnorm(100,78,10). How does a situation like this change your assumptions on either $y_i$ or $\epsilon_i$? –  PeterRabbit Jan 28 at 11:36
    
In that case, heteroscedasticity would be a problem and you would need to use some other form of regression, or possibly some transformation, or you could add another variable (in this silly example, position played in basketball might do it). –  Peter Flom Jan 28 at 13:14
    
I'm not sure the formulation is intended to suggest that the ys are normally distributed, just that they have a normal conditional distribution. –  Dikran Marsupial Jan 28 at 16:15
add comment

You need to add a suscripted i to your second formulation:
$$ y_i\sim\mathcal N(\hat y_i,\sigma^2_\varepsilon) $$ because $\hat y$ needs to be able to vary along with $\bf x_i$.


That having been noted, what is $\hat y_i$? It is $\bf x_i\boldsymbol{\hat\beta}$. This leads to the formulation @DikranMarsupial presents:
$$ y_i\sim\mathcal N({\bf x_i}\boldsymbol{\hat\beta},\sigma^2_\varepsilon) $$ It is worth recognizing that this is exactly the same as your first formulation, because both stipulate normal distributions and the expected values are equal. That is:
\begin{align} E[{\bf x_i}\boldsymbol{\hat\beta}] &= E[{\bf x_i}\boldsymbol{\hat\beta} + E[\mathcal N(0, \sigma^2_\varepsilon)]] \\ &= E[{\bf x_i}\boldsymbol{\hat\beta} + 0] \\ &= E[{\bf x_i}\boldsymbol{\hat\beta}] \end{align} (And obviously the variances are equal.) In other words, this is not a difference in assumptions, but simply a notational difference.

So the question becomes, is there a reason to prefer presenting the idea using the first formulation?

I think the answer is yes for two reasons:

  1. People often confuse whether the raw data should be normally distributed (i.e., $Y$), or if the data conditional on $\bf X$ / the errors should be normally distributed (i.e., $Y|\bf X$ / $\varepsilon$), for example, see: What if residuals are normally distributed, but y is not?
  2. People also often confuse what is supposed to be independent, the raw data or the errors. Moreover, we often mention the fact that something should be iid (independent and identically distributed); if you are thinking in terms of $Y|\bf X$ this can be another potential source of confusion, as $Y|\bf X$ can be independent, but cannot be identically distributed unless the null hypothesis holds (because the mean would vary).

I believe these confustions are more likely using the second formulation than the first.

share|improve this answer
1  
@Glen_b, I don't follow your comment. My claim is not that $\hat y$ is equal to $X\beta$, but rather that $\hat y_i$ is equal to $\bf x_i\boldsymbol{\hat\beta}$. The subscripted $_i$ indexing the observations is relevant. The idea is that the predicted value, $\hat y_i$, for a given observation is $\bf x_i\boldsymbol{\hat\beta}$. This has nothing to do w/ the population mean of $Y$. (It appears that I had forgotten to add hats to my betas, though; I've corrected that now.) –  gung Jan 28 at 22:38
    
@Glen_b if it were the sample mean it would be $\bar{y}$ rather than $\hat{y}$. I initially found the notation confusing as well, but the fact that $\hat{y} = X\beta$ follows from the statements that $y_i = X\beta + \epsilon_i$ and $\epsilon_i = y_i - \hat{y}$. For both of these things to be true, $\hat{y}$ can only be $X\beta$. –  Dikran Marsupial Jan 29 at 8:29
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.