43
$\begingroup$

What is the main difference between maximum likelihood estimation (MLE) vs. least squares estimaton (LSE) ?

Why can't we use MLE for predicting $y$ values in linear regression and vice versa?

Any help on this topic will be greatly appreciated.

$\endgroup$
  • 8
    $\begingroup$ You can use MLE in linear regression if you like. This can even make sense if the error distribution is non-normal and your goal is to obtain the "most likely" estimate rather than one which minimizes the sum of squares. $\endgroup$ – Richard Hardy Mar 27 '15 at 14:57
  • 16
    $\begingroup$ Under normal error assumption, as is typically assumed in linear regression, the MLE and the LSE are the same! $\endgroup$ – TrynnaDoStat Mar 27 '15 at 15:07
  • 1
    $\begingroup$ Search our site for the Gauss-Markov theorem. $\endgroup$ – whuber Mar 27 '15 at 17:41
  • $\begingroup$ thanks for all the replies. Now this makes sense. While searching for this topic on the net, I came across this article. Maybe this also helps:radfordneal.wordpress.com/2008/08/09/… $\endgroup$ – evros Mar 27 '15 at 19:17
  • 1
    $\begingroup$ An answer is also provided at stats.stackexchange.com/questions/12562/…. $\endgroup$ – whuber Dec 12 '15 at 21:40
20
$\begingroup$

I'd like to provide a straightforward answer.

What is the main difference between maximum likelihood estimation (MLE) vs. least squares estimation (LSE) ?

As @TrynnaDoStat commented, minimizing squared error is equivalent to maximizing the likelihood in this case. As said in Wikipedia,

In a linear model, if the errors belong to a normal distribution the least squares estimators are also the maximum likelihood estimators.

they can be viewed as the same in your case,

Let me detail it a bit. Since we know that the response variable($y$) $$Y_i=\lambda_1X_i+\lambda_2+\epsilon_i \quad\text{ where }\epsilon\thicksim N(0,\sigma^2)$$ has a normal error distribution model,
enter image description here
the likelihood function is, $$L(Y_1,\dots,Y_n;\lambda_1,\lambda_2,\sigma^2)=\frac{1}{(2\pi)^{\frac{n}{2}\sigma^n}}exp(\frac{-1}{2\sigma^2}(\sum_{i=1}^n(Y_i-\lambda_1X_i-\lambda_2)^2))$$ Obviously maximizing L is equivalent to minimizing $$\sum_{i=1}^n(Y_i-\lambda_1X_i-\lambda_2)^2$$ That's the least squares method.

Why can't we use MLE for predicting $y$ values in linear regression and vice versa?

As explained above we're actually(more precisely equivalently) using the MLE for predicting $y$ values. And if the response variable has arbitrary distributions rather than normal distribution, like Bernoulli distribution or any one from the exponential family we map the linear predictor to the response variable distribution using a link function(according to the response distribution), then the likelihood function becomes the product of all the outcomes(probabilities between 0 and 1) after the transformation. We can treat the link function in the linear regress as the identity function(since the response is already a probability).

$\endgroup$
  • 3
    $\begingroup$ You may want to define "this case" a bit more clearly since in general, maximum likelihood and least squares are not the same thing. $\endgroup$ – Matthew Gunn Mar 4 '17 at 18:48
  • 2
    $\begingroup$ @MatthewGunn Yeah, I used "equivalent to" other than "the same". $\endgroup$ – Lerner Zhang Mar 5 '17 at 1:29
  • $\begingroup$ Would be great if you would give us an example where the linear model follows non-normal error distribution, and how you use MLE in such a case to estimate the best coefficients. If not possible, at least can you point us to a correct source, which demonstrates this using linear models like Poisson regression $\endgroup$ – VM_AI May 30 at 12:20
12
$\begingroup$

ML is a higher set of estimators which includes least absolute deviations ($L_1$-Norm) and least squares ($L_2$-Norm). Under the hood of ML the estimators share a wide range of common properties like the (sadly) non-existent break point. In fact you can use the ML approach as a substitute to optimize a lot of things including OLS as long as you are aware what you're doing.

$L_2$-Norm goes back to C. F. Gauss and is around 200 years old while the modern ML approach goes back to (IMHO) Huber 1964. Many scientists are used to $L_2$-Norms and their equations. The theory is well understood and there are a lot of published papers which can be seen as useful extensions like:

  • data snooping
  • stochastic parameters
  • weak constraints

Professional applications don't just fit data, they check:

  • if the parameter are significant
  • if your dataset has outliers
  • which outlier can be tolerated since it does not cripple the performance
  • which measurement should be removed since it does not contribute to the degree of freedoms

Also there are huge number of specialized statistic tests for hypotheses. This does not necessary apply to all ML estimators or should be at least stated with a proof.

Another profane point is that $L_2$-Norm is very easy to implement, can be extended to Bayesian regularization or other algorithms like Levenberg-Marquard.

Not to forget: Performance. Not all least square cases like Gauss-Markov $\mathbf{X\beta}=\mathbf{L}+\mathbf{r}$ produce symmetric positive definite normal equations $(\mathbf{X}^{T}\mathbf{X})^{-1}$. Therefore I use a separate libraries for each $L_2$-Norm. It is possible to perform special optimizations for this certain case.

Feel free to ask for details.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.