14
$\begingroup$

My question is very simple: why we choose normal as the distribution that error term follows in the assumption of linear regression? Why we don't choose others like uniform, t or whatever?

$\endgroup$
  • 5
    $\begingroup$ We don't choose the normal assumption. It just happens to be the case that when the error is normal, the model coefficients exactly follow a normal distribution and an exact F-test can be used to test hypotheses about them. $\endgroup$ – AdamO Mar 1 at 4:25
  • 10
    $\begingroup$ Because the math works out easily enough that people could use it before modern computers. $\endgroup$ – Nat Mar 1 at 4:32
  • 1
    $\begingroup$ @AdamO I don't understand; you just outlined the reasons why we choose it. $\endgroup$ – JiK Mar 1 at 14:34
  • 2
    $\begingroup$ @JiK if I could choose distributions, there'd be no need for statistics at all. The whole world would be probability. $\endgroup$ – AdamO Mar 1 at 15:40
  • 1
    $\begingroup$ @AdamO You can choose assumptions for your model when you're doing statistical inference, so I don't think that means there is no statistics. $\endgroup$ – JiK Mar 1 at 21:45
29
$\begingroup$

We do choose other error distributions. You can in many cases do so fairly easily; if you are using maximum likelihood estimation, this will change the loss function. This is certainly done in practice.

Laplace (double exponential errors) correspond to least absolute deviations regression/$L_1$ regression (which numerous posts on site discuss). Regressions with t-errors are occasionally used (in some cases because they're more robust to gross errors), though they can have a disadvantage -- the likelihood (and therefore the negative of the loss) can have multiple modes.

Uniform errors correspond to an $L_\infty$ loss (minimize the maximum deviation); such regression is sometimes called Chebyshev approximation (though beware, since there's another thing with essentially the same name). Again, this is sometimes done (indeed for simple regression and smallish data sets with bounded errors with constant spread the fit is often easy enough to find by hand, directly on a plot, though in practice you can use linear programming methods, or other algorithms; indeed, $L_\infty$ and $L_1$ regression problems are duals of each other, which can lead to sometimes convenient shortcuts for some problems).

In fact, here's an example of a "uniform error" model fitted to data by hand:

L-infinity regression fitted by hand. The two "lowest" points under the strip of data points are marked, and the two "highest" points above the strip of data are marked.

It's easy to identify (by sliding a straightedge toward the data) that the four marked points are the only candidates for being in the active set; three of them will actually form the active set (and a little checking soon identifies which three lead to the narrowest band that encompassess all the data). The line at the center of that band (marked in red) is then the maximum likelihood estimate of the line.

Many other choices of model are possible and quite a few have been used in practice.

Note that if you have additive, independent, constant-spread errors with a density of the form $k\,\exp(-c.g(\varepsilon))$, maximizing the likelihood will correspond to minimizing $\sum_i g(e_i)$, where $e_i$ is the $i$th residual.

However, there are a variety of reasons that least squares is a popular choice, many of which don't require any assumption of normality.

$\endgroup$
  • 2
    $\begingroup$ Great answer. Would you mind adding some links which give more details as to how these variations are used in practice? $\endgroup$ – rgk Mar 1 at 18:52
  • $\begingroup$ (+1) Great answer. Would you mind sharing the R-code used for fitting the $L_{\infty}$-Regression line? $\endgroup$ – COOLSerdash Mar 2 at 10:35
  • 1
    $\begingroup$ As I explained in the text, I fitted it by hand, in a very similar fashion to the approach I described. While it can be done readily enough using code, I literally opened the plot in MS Paint and identified the three points in the active set (joining two of which gave the slope) -- and then moved the line half-way toward the third point (by halving the vertical distance in pixels and moving the line up that many pixels) -- the point being to demonstrate quite how simple this could be. A child could be taught to do it. $\endgroup$ – Glen_b Mar 2 at 10:39
  • $\begingroup$ @Glen_b Indeed, I was a teenager when I was taught to do exactly that in freshman physics lab. $\endgroup$ – Peter Leopold Mar 2 at 14:10
9
$\begingroup$

The normal/Gaussian assumption is often used because it is the most computationally convenient choice. Computing the maximum likelihood estimate of the regression coefficients is a quadratic minimization problem, which can be solved using pure linear algebra. Other choices of noise distributions yield more complicated optimization problems which typically have to be solved numerically. In particular, the problem may be non-convex, yielding additional complications.

Normality is not necessarily a good assumption in general. The normal distribution has very light tails, and this makes the regression estimate quite sensitive to outliers. Alternatives such as the Laplace or Student's t distributions are often superior if measurement data contain outliers.

See Peter Huber's seminal book Robust Statistics for more information.

$\endgroup$
2
$\begingroup$

When working with those hypothesis, squared-erros based regression and maximum likelihood provide you the same solution. You are also capable of getting simple F-tests for coefficient significance, as well as confidence intervals for your predictions.

In conclusion, the reason why we often choose normal distribution is its properties, which often make things easy. It is also not a very restrictive assumption, as many other types of data will behaive "kind-of-normally"

Anyway, as mentioned in a previous answer, there are possibilities to define regression models for other distributions. The normal just happens to be the most recurrent one

$\endgroup$
2
$\begingroup$

Glen_b has explained nicely that OLS regression can be generalized (maximizing likelihood instead of minimizing sum of squares) and we do choose other distributions.

However, why is the normal distribution chosen so often?

The reason is that the normal distribution occurs in many places naturally. It is a bit the same like we often see the golden ratio or the Fibonacci numbers occurring "spontaneously" at various places in nature.

The normal distribution is the limiting distribution for a sum of variables with finite variance (or less strict restrictions are possible as well). And, without taking the limit, it is also a good approximation for a sum of a finite number of variables. So, because many observed errors occur as a sum of many little unobserved errors, the normal distribution is a good approximation.

See also here Importance of normal distribution

where Galton's bean machines show the principle intuitively

https://upload.wikimedia.org/wikipedia/commons/7/7f/Quincunx_%28Galton_Box%29_-_Galton_1889_diagram.png

$\endgroup$
-1
$\begingroup$

Why we don't choose other distributions?—we do.

Regression means modeling a continuous values given a set of inputs. Consider training examples consisting of a target scalar $y_i \in \mathbb R$ and an input vector $x_i \in \mathbb R^n$. Let the prediction of the target given $x_i$ be

$$\hat y_i = w^\intercal x_i.$$

The surprisal loss is usually the most sensible loss:

$$L = -\log P(y_i \mid x_i).$$

You can think of linear regression as using a normal density with fixed variance in the above equation:

$$L = -\log P(y_i \mid x_i) \propto (y_i - \hat y_i)^2.$$

This leads to the weight update:

$$\nabla_w L = (\hat y_i - y_i)x_i $$


In general, if you use another exponential family distribution, this model is called a generalized linear model. The different distribution corresponds to a different density, but it can be more easily formalized by changing the prediction, the weight, and the target.

The weight is changed to a matrix $W \in \mathbb R^{n\times k}$. The prediction is changed to

$$\hat u_i \triangleq \nabla g(W x_i)$$

where $\nabla g: \mathbb R^k \to \mathbb R^k$ is called the link function or gradient log-normalizer. And, the target $y_i$ is changed to a vector called sufficient statistics $u_i = T(y_i) \in \mathbb R^k$.

Each link function and sufficient statistics corresponds to a different distributional assumption, which is what your question is about. To see why, let's look at a continuous-valued exponential family's density function with natural parameters $\eta$:

$$f(z) = h(z)\exp(\eta^\intercal T(z) - g(\eta)).$$

Let the natural parameters $\eta$ be $w^\intercal x_i$, and evaluate the density at the observed target $z = y_i$. Then, the loss gradient is

$$\begin{align} \nabla_W L &= \nabla_W -\log f(x) \\ &= (\nabla g(W x_i)) x_i^\intercal - T(y_i) x_i^\intercal \\ &= (\hat u_i - u_i) x_i^\intercal \end{align},$$ which has the same nice form as linear regression.


As far as I know, the gradient log-normalizer can be any monotonic, analytic function, and any monotonic, analytic function is the gradient log-normalizer of some exponential family.

$\endgroup$
  • $\begingroup$ This is very short and too cryptic for our standards, please also explain surprisal. $\endgroup$ – kjetil b halvorsen Mar 2 at 15:27
  • 1
    $\begingroup$ "each link function corresponds to a different distributional assumption" this is very vague. The link function does not have to do with generalizing to different distributional assumptions, but with generalizing the (linear) part that describes the mean of the distribution. $\endgroup$ – Martijn Weterings Mar 3 at 20:57
  • 1
    $\begingroup$ The linked article contains in section '3.1 Normal distribution' > "More generally, as shown in Nelder (1968), we can consider models in which there is a linearizing transformation $f$ and a normalizing transformation $g$" I do not know what your gradient log-normalizer refers to, and maybe you are speaking about this normalizing transformation? But, that is not the link function. The link function in GLM relates to the linearizing transformation. $\endgroup$ – Martijn Weterings Mar 3 at 22:13
  • 1
    $\begingroup$ Typically certain link functions are used with certain distributional assumptions. But this is not a necessity. So my distributional assumptions are normal in that example, and not Poisson (that was intentional). Some better (more practical and well known) examples are binomial/Bernouilli distributed variables where people work with a probit model or a logit model, thus different link functions but the same (conditional) distributional assumption. $\endgroup$ – Martijn Weterings Mar 3 at 22:40
  • 1
    $\begingroup$ @Neil G: I'm the lazy one? You could easily have included surprisal in the original post, yes? Also, when I am making such comments, is is more for the site than for myself. This site is supposed to be self-contained. I could have/did guess the meaning (even if it is nonstandard terminology in statistics), as you can see from my answer here, entropy $\endgroup$ – kjetil b halvorsen Mar 17 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.