26
$\begingroup$

In linear regression, each predicted value is assumed to have been picked from a normal distribution of possible values. See below.

But why is each predicted value assumed to have come from a normal distribution? How does linear regression use this assumption? What if possible values are not normally distributed?

enter image description here

$\endgroup$
  • 2
    $\begingroup$ Only the errors follow a normal distribution (which implies the conditional probability of Y given X is normal too). This is probably traditional because of reasons relating to the central limit theorem. But you can replace normal with any symmetric probability distribution and get the same estimates of coefficients via least squares. What differs though would be the residual standard error, goodness of fit and the way you validate the assumptions. $\endgroup$ – Kian Apr 29 '15 at 9:15
  • 4
    $\begingroup$ Normal assumptions mainly come into inference -- hypothesis testing, CIs, PIs. If you make different assumptions, those will be different, at least in small samples. $\endgroup$ – Glen_b -Reinstate Monica Apr 29 '15 at 10:20
  • 7
    $\begingroup$ Incidentally, for ordinary linear regression your diagram should draw the normal curves vertically, not diagonally. $\endgroup$ – Glen_b -Reinstate Monica Apr 29 '15 at 11:02
29
$\begingroup$

Linear regression by itself does not need the normal (gaussian) assumption, the estimators can be calculated (by linear least squares) without any need of such assumption, and makes perfect sense without it.

But then, as statisticians we want to understand some of the properties of this method, answers to questions such as: are the least squares estimators optimal in some sense? or can we do better with some alternative estimators? Then, under the normal distribution of error terms, we can show that this estimators are, indeed, optimal, for instance they are "unbiased of minimum variance", or maximum likelihood. No such thing can be proved without the normal assumption.

Also, if we want to construct (and analyze properties of) confidence intervals or hypothesis tests, then we use the normal assumption. But, we could instead construct confidence intervals by some other means, such as bootstrapping. Then, we do not use the normal assumption, but, alas, without that, it could be we should use some other estimators than the least squares ones, maybe some robust estimators?

In practice, of course, the normal distribution is at most a convenient fiction. So, the really important question is, how close to normality do we need to be to claim to use the results referred to above? That is a much trickier question! Optimality results are not robust, so even a very small deviation from normality might destroy optimality. That is an argument in favour of robust methods. For another tack at that question, see my answer to Why should we use t errors instead of normal errors?

Another relevant question is Why is the normality of residuals "barely important at all" for the purpose of estimating the regression line?

 EDIT

This answer led to a large discussion-in-comments, which again led to my new question: Linear regression: any non-normal distribution giving identity of OLS and MLE? which now finally got (three) answers, giving examples where non-normal distributions lead to least squares estimators.

$\endgroup$
  • $\begingroup$ Least squares error is equivalent to a normal assumption. $\endgroup$ – Neil G Apr 29 '15 at 9:04
  • 4
    $\begingroup$ There is no such contradiction. For instance, the Gauss-Markov theorem says that linear least squares is optimal (in least variance sense) among all linear estimators, without any need of distributional assumptions (apart from existing variance). Least squares is a numerical procedure which can be defined independent of any probabilistic model! The probabilistic model is then used to analyze this procedure from a statistical perspective. $\endgroup$ – kjetil b halvorsen Apr 29 '15 at 9:19
  • 2
    $\begingroup$ @NeilG Certainly MLE for the normal is least squares but that doesn't imply least squares must entail an assumption of normality. On the other hand, large deviations from normality may make least squares a poor choice (when all linear estimators are bad). $\endgroup$ – Glen_b -Reinstate Monica Sep 19 '15 at 23:29
  • 1
    $\begingroup$ @NeilG What I said there doesn't in any way imply equivalence of LS and normality, but you say explicitly they are equivalent, so I really don't think our two statements are even close to tautological. $\endgroup$ – Glen_b -Reinstate Monica Sep 20 '15 at 16:11
  • 1
    $\begingroup$ @Neil Can you show how your statement actually implies what I said? I really don't see it. $\endgroup$ – Glen_b -Reinstate Monica Sep 21 '15 at 5:43
3
$\begingroup$

This discussionWhat if residuals are normally distributed, but y is not? has well addressed this question.

In short, for a regression problem, we only assume that the response is normal conditioned on the value of x. It is not necessary that the independent or response variables are independent.

$\endgroup$
1
$\begingroup$
  1. But why is each predicted value assumed to have come from a normal distribution?

There is no deep reason for it, and you are free to change the distributional assumptions, moving to GLMs, or to robust regression. The LM (normal distribution) is popular because its easy to calculate, quite stable and residuals are in practice often more or less normal.

  1. How does linear regression use this assumption?

As any regression, the linear model (=regression with normal error) searches for the parameters that optimize the likelihood for the given distributional assumption. See here for an example of an explicit calculation of the likelihood for a linear model. If you take the log likelihood of a linear model, it turns out to be proportional to the sum of squares, and the optimization of that can be calculated quite conveniently.

  1. What if possible values are not normally distributed?

If you want to fit a model with different distributions, the next textbook steps would be generalized linear models (GLM), which offer different distributions, or general linear models, which are still normal, but relax independence. Many other options are possible. If you just want to reduce the effect of outliers, you could for example consider robust regression.

$\endgroup$
0
$\begingroup$

After review the question again, I think there is no reason to use the normal distribution unless you want to perform some kind of inference about the parameter of regression. And you can apply linear regression and ignore the distribution of noise term.

$\endgroup$
  • 2
    $\begingroup$ Doesn't make much of a sense to me. $\endgroup$ – SmallChess May 17 '15 at 9:35
0
$\begingroup$

Let me stick to the case of a one variable regression. The details are the same, but the notation is more cumbersome in the case of a multivariate regression. Given any data set $(x_i,y_i)$ one can find the 'least squares line' $ y = \beta x +c$ , that is find $\beta$ so that $\sum_i (y_i - \sum_i \beta x_i - c)^2$ is minimized. That is pure mathematics. However under the assumption that the residuals $ \eta_i = y_i - (\beta x_i +c) $ are independent identically distributed gaussian variables with a common variance, then one can get statistical estimates of how accurate the point estimate $\beta$. In particular, one can construct the 95% confidence interval for $\beta$. After all we are assuming that we are sampling from the underlying (true) distribuion and hence if we sampled again, we should expect to get a, possibly just slightly, different answer. In particular, the p-value is the probability of observing the given $\beta$ under the hypothesis that the true value of $\beta$ is zero. So the statistics comes about as information about how accurate is the point estimate $\beta$ . What to do in the case one doesn't have statistical properties of the error term ? With apologies to "The Graduate" - one word bootstrap.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.