14
$\begingroup$

So when I assume that the error terms are normally distributed in a linear regression, what does it mean for the response variable, $y$?

$\endgroup$
7
$\begingroup$

Maybe I'm off but I think we ought to be wondering about $f(y|\beta, X)$, which is how I read the OP. In the very simplest case of linear regression if your model is $y=X\beta + \epsilon$ then the only stochastic component in your model is the error term. As such it determines the sampling distribution of $y$. If $\epsilon\sim N(0, \sigma^2I)$ then $y|X, \beta\sim N(X\beta, \sigma^2I)$. What @Aniko says is certainly true of $f(y)$ (marginally over $X, \beta$), however. So as it stands the question is slightly vague.

$\endgroup$
  • $\begingroup$ I like all comments! And they all seem to be right. But I was just searching for the easiest answer :) What happens when you assume that the errer term is normal distributed. That this occurs now very often in reality gets clear from the other answers! Thanks a lot! $\endgroup$ – MarkDollar May 29 '11 at 7:57
17
$\begingroup$

The short answer is that you cannot conclude anything about the distribution of $y$, because it depends on the distribution of the $x$'s and the strength and shape of the relationship. More formally, $y$ will have a "mixture of normals" distribution, which in practice can be pretty much anything.

Here are two extreme examples to illustrate this:

  1. Suppose there are only two possible $x$ values, 0 an 1, and $y = 10x + N(0,1)$. Then $y$ will have a strongly bimodal distribution with bumps at 0 and 10.
  2. Now assume the same relationship, but let $x$ be uniformly distributed on the 0-1 interval with lots of values. Then $y$ will be almost uniformly distributed over the 0-10 interval (with some half-normal tails at the edges).

In fact, since every distribution can be approximated arbitrarily well with mixture of normals, you can really get any distribution for $y$.

$\endgroup$
  • 8
    $\begingroup$ +1 Re the last statement: I once made the mistake of thinking that too. Mathematically you're correct but in practice it's nearly impossible to approximate a non-differentiable spike with normals (such as J- or U-shaped distributions): the normals are just too flat at their peaks to capture the density in the spikes. You need way too many components. Normals are good for approximating distributions whose pdfs are very smooth. $\endgroup$ – whuber May 27 '11 at 17:01
  • 1
    $\begingroup$ @whuber Agreed. I would not suggest using a normal-mixture approximation for any distribution in practice, I was just trying to give an extreme counter-example. $\endgroup$ – Aniko May 27 '11 at 20:33
5
$\begingroup$

We invent the error term by imposing a fictitious model on real data; the distribution of the error term does not affect the distribution of the response.

We often assume that the error is distributed normally and thus try to construct the model such that our estimated residuals are normally distributed. This can be difficult for some distributions of $y$. In these cases, I suppose you could say that the distribution of the response affects the error term.

$\endgroup$
  • 2
    $\begingroup$ "We often try to construct the model such that our error term is normally distributed" - to be precise, I think you are referring to the residuals $y-X\hat\beta$. These are estimates of the error terms in the same way that $X\hat\beta$ is an estimate of $\mathbb{E}(y)=X\beta$. We'd like the residuals to look normal because that's what we assumed about the error terms to begin with. We "invent" the error term by specifying a model, not fitting it. $\endgroup$ – JMS May 28 '11 at 17:01
  • $\begingroup$ I agree with your precision, JMS. +1 and I'll adjust my answer. $\endgroup$ – Thomas Levine May 28 '11 at 19:02
2
$\begingroup$

If you write out the response as $$\bf{y}=m+e$$ Where $\bf{m}$ is the "model" (the prediction for $\bf{y}$) and $\bf{e}$ is the "errors", then this can be re-arranged to indicate $\bf{y}-m=e$. So assigning a distribution for the errors is the same thing as indicating the ways your model is incomplete. To put it another way is that it indicates to what extent you don't know why the observed response was the value that it actually was, and not what the model predicted. If you knew your model was perfect, then you would assign a probability distribution with all of its mass on zero for the errors. Assigning a $N(0,\sigma^{2})$ basically says that the errors are small in units of $\sigma$. The idea is that the model predictions tend to be "wrong" by similar amounts for different observations, and is "about right" on the scale of $\sigma$. As a contrast, an alternative assignment is $Cauchy(0,\gamma)$ which says that most of the errors are small, but some errors are quite large - the model has the occasional "blunder" or "shocker" in terms of predicting the response.

In a sense the error distribution is more closely linked to the model than to the response. This can be seen from the non-identifiability of the above equation, for if both $\bf{m}$ and $\bf{e}$ are unknown then adding an arbitrary vector to $\bf{m}$ and subtracting it from $\bf{e}$ leads to the same value of $\bf{y}$, $\bf{y}=m+e=(m+b)+(e-b)=m'+e'$. The assignment of an error distribution and a model equation basically says which arbitrary vectors are more plausible than others.

$\endgroup$
  • $\begingroup$ "This seems strange because you will only observe y once and only once (y is the complete vector/matrix/etc. of responses). How can this be "distributed"? In my view it can only be distributed in some imaginary ensemble, nothing to do with your actual observed response. At the very least, any such presumption of the response "being distributed" is untestable" I'm confused; are you saying we can't test $H_0: y\sim f_0$ vs $H_1: y\sim f_1$? $\endgroup$ – JMS May 28 '11 at 17:07
  • $\begingroup$ no, sorry, that can't be what you're saying. I'm still confused though. Maybe it's slightly imprecise, but the way I read it he's got $n$ samples of $y_i$ from $Y$ with fixed $x_i$, his model is $Y = X\beta + \epsilon$, and he's wondering what the assumed distribution of $\epsilon$ implies about the distribution of $Y|\beta, X$ under his model. Here it would imply that it's normal; we can test that with our sample $\endgroup$ – JMS May 28 '11 at 17:21
  • $\begingroup$ @JMS - I think I might delete that first paragraph. I don't think it adds anything to my answer (besides confusion). $\endgroup$ – probabilityislogic May 28 '11 at 23:33
  • $\begingroup$ one of my favorite things to add to my answers :) $\endgroup$ – JMS May 29 '11 at 2:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.