11
$\begingroup$

Title says it all. I understand that the Least-Squares and Maximum-Likelihood will give the same result for regression coefficients if the model's errors are normally distributed. But, what happens if the errors are not normally distributed? Why are the two methods no longer equivalent?

$\endgroup$
  • $\begingroup$ Do you mean (a) using MLE when the assumption of normality is not met, or (b) using non-Gaussian likelihood function? $\endgroup$ – Tim Dec 7 '17 at 15:02
  • $\begingroup$ (a), when the assumption of normality is not met $\endgroup$ – Shuklaswag Dec 7 '17 at 19:24
  • $\begingroup$ Even when the assumption is not met (ie the observed values are not Gaussian distributed)... if you calculate the MLE with the use of the Gaussian likelihood function then you do the same as least squares optimization. The optimization methods are mathematically equivalent, and independent from whether the assumption of normality was right or not. $\endgroup$ – Sextus Empiricus Dec 8 '17 at 0:31
  • $\begingroup$ Even with normal distributions, least-squares imposes fixed variance. $\endgroup$ – CodesInChaos Dec 8 '17 at 8:53
  • $\begingroup$ See also this related question: stats.stackexchange.com/questions/173621/… $\endgroup$ – kjetil b halvorsen Dec 8 '17 at 13:52
17
$\begingroup$

Short Answer

The probability density of a multivariate Gaussian distributed variable $x=(x_1, x_2,...,x_n)$, with mean $\mu=(\mu_1,\mu_2,...,\mu_n)$ is related to the square of the euclidean distance between the mean and the variable ($\vert \mu-x \vert_2^2$), or in other words the sum of squares.


Long Answer

If you multiply multiple Gaussian distributions for your $n$ errors, where you assume equal deviations, then you get a sum of squares.

$$ \begin{array} \mathcal{L(\mu_j,x_{ij})} = P(x_{ij} \vert \mu_j) & =\prod_{i=1}^n \frac{1}{\sqrt{2 \pi \sigma^2}} exp\left[-\frac{(x_{ij}-\mu_i)^2}{2\sigma^2}\right] \\ &= \left(\frac{1}{\sqrt{2 \pi \sigma^2}} \right)^n exp \left[ -\frac{\sum_{i=1}^n(x_{ij}-\mu_i)^2}{2\sigma^2}\right] \end{array}$$

or in the convenient logarithmic form:

$$ \log\left(\mathcal{L(\mu_j,x_{ij})} \right) = n \log \left( \frac{1}{\sqrt{2 \pi \sigma^2}} \right) -\frac{1}{2\sigma^2} \sum_{i=1}^n(x_{ij}-\mu_j)^2 $$

So optimizing the $\mu$ to minimize the sum of squares is equal to maximizing the (log) likelihood (ie. the product of multiple Gaussian distributions, or the multivariate Gaussian distribution).

It is this nested square of the difference $(\mu-x)$ inside exponential structure, $exp\left[ (x_i-\mu)^2 \right]$, which other distributions do not have.


Compare for instance with the case for Poisson distributions

$$\log(\mathcal{L}) = \log \left( \prod\frac{\mu_j^{x_{ij}}}{x_{ij}!} exp \left[ -\mu_j \right] \right) = -\sum \mu_j -\sum log(x_{ij}!) + \sum log(\mu_j) x_{ij} $$

which has a maximum when the following is minimized:

$$\sum \mu_j -log(\mu_j) x_{ij}$$

which is a different beast.


In addition (history)

The history of the normal distribution (ignoring deMoivre getting to this distribution as an approximation for the binomial distribution) is actually as the discovery of the distribution that makes the MLE correspond to the least squares method (rather than the the least squares method being a method that can express the MLE of the normal distribution, first came the least squares method, second came the Gaussian distribution)

Note that Gauss, connecting the 'method of maximum likelihood' with the 'the method of least squares', came up with the 'Gaussian distribution', $e^{-x^2}$ , as the sole distribution of errors that leads us to make this connection between the two methods.

From Charles Henry Davis' translation (Theory of the motion of the heavenly bodies moving about the sun in conic sections. A translation of Gauss's "Theoria motus," with an appendix) ...

Gauss defines:

Accordingly, the probability to be assigned to each error $\Delta$ will be expressed by a function of $\Delta$ which we shall denote by $\psi \Delta$.

(Italization done by me)

And continues (in section 177 pp. 258):

... whence it is readily inferred that $\frac{\psi^\prime\Delta}{\Delta}$ must be a constant quantity. which we will denote by $k$. Hence we have $$\text{log } \psi \Delta = \frac{1}{2} k \Delta \Delta + \text{Constant}$$ $$\psi \Delta = x e^{\frac{1}{2}k \Delta \Delta}$$ denoting the base of the hyperbolic logarithms by $e$ and assuming $$\text{Constant} = \log x$$

ending up (after normalization and realizing $k<0$) in

$$\psi \Delta = \frac{h}{\sqrt{\pi}} e^{-hh\Delta \Delta} $$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Do you remember where you got this knowledge from? Would you mind adding the source to your post? (I'm having a hard time finding a textbook that explains this well.) $\endgroup$ – Joooeey Jan 8 '19 at 22:15
  • $\begingroup$ @Joooeey I have added the title of the source for the translated quotes of Gauss as well as a link to one of many online sources. This original text is heavy, but you should encounter lighter treaties in any description of the history of the normal distribution. $\endgroup$ – Sextus Empiricus Jan 9 '19 at 0:33
  • $\begingroup$ The likelihood functions are popping up in many places. If you look for sources where I got this 'knowledge' then I guess I could say Pearson's 1900 article about the chi-squared test where the multivariate normal distribution is treated geometrically. Also Fisher used geometrical representations several times (there is for instance this one article in the 20s , about efficiency of estimates, where he compares the mean squared error and mean absolute error and where he speaks about surfaces in a hyperspace). $\endgroup$ – Sextus Empiricus Jan 9 '19 at 0:38
  • $\begingroup$ @Joooeey I have made a reference to that Fisher article before here. And my answer here uses a geometrical viewpoint to derive a property of the t-distribution relates to Fisher as well (I believe the article where he proofs Gosset's t-distribution or maybe some slightly later article). $\endgroup$ – Sextus Empiricus Jan 9 '19 at 0:52
5
$\begingroup$

Because the MLE is derived from the assumption of residual normally distributed.

Note that

$$ \text{min}_\beta~~ \|X \beta - y \|^2 $$

Has no probabilistic meaning: just find the $\beta$ that minimize the squared loss function. Everything is deterministic, and no random components in there.

Where the concept of probability and likelihood come, is we assume

$$ y=X\beta + \epsilon $$

Where we are considering $y$ as a random variable, and $\epsilon$ is normally distributed.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ @Matthew Drury why change the matrix notation and add sum sign? $\endgroup$ – Haitao Du Dec 7 '17 at 15:52
  • $\begingroup$ I figured it would be clear, but if your claiming that a statement has no probalistic meaning, you cant use an expression with symbols that are best interpreted as random variables. The optimaziation problem you are refencing is in relation to fixed data, i made that explicit. $\endgroup$ – Matthew Drury Dec 7 '17 at 16:44
5
$\begingroup$

The least squares and the maximum (gaussian) likelihood fit are always equivalent. That is, they are minimized by the same set of coefficients.

Changing the assumption on the errors changes your likelihood function (maximizing the likelihood of a model is equivalent to maximizing the likelihood of the error term), and hence the function will no longer be minimized by the same set of coefficients.

So in practice the two are the same, but in theory, when you maximize a different likelihood, you will get to a different answer than Least-squares

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ "or always equivalent"? $\endgroup$ – nbro Jun 16 '18 at 16:23
1
$\begingroup$

A concrete example: Suppose we take a simple error function p(1)=.9, p(-9) =.10 . If we take two points, then LS is just going to take the line through them. ML, on the other hand, is going to assume that both points are one unit too high, and thus will take the line through the points shifted down on unit.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Your example is unclear; in particular, it's difficult to see what model you are trying to describe or why ML would produce the result you claim. Could you elaborate further in this answer? $\endgroup$ – whuber Dec 7 '17 at 22:50
  • $\begingroup$ The model is that y=mx+b+error, where error has a 90% chance of being +1 and 10% chance of being -9. Given any observed point, the true point has a 90% likelihood of being one unit below and a 10% likelihood of being nine units above. Therefore, ML gives that the true point is one unit below. What don't you understand about this? $\endgroup$ – Acccumulation Dec 7 '17 at 23:08
  • 2
    $\begingroup$ Your comment is helpful, but your answer still doesn't describe the model in any clear or understandable way. Could you incorporate that explanation in the answer itself? It's a nice example. $\endgroup$ – whuber Dec 7 '17 at 23:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.