Forgive my very limited understanding. I am trying to learn about maximum likelihood estimation, and how it differs from least-squares estimation. From reading a little, I understand that the two are equivalent when the errors are Gaussian-distributed (see this question and this question for example).

Could anyone provide a concrete example to demonstrate a case when the errors are non-Gaussian that demonstrates LSE failing, and instead MLE correctly giving back the true parameter?

I am basically looking for a practical example which demonstrates when MLE is able to solve a problem that LSE cannot, but I can't visualise such an example for myself at the moment. Something simple, at the measuring-the-length-of-my-desk type of level.

I hope my question is clear enough, thank you!

  • $\begingroup$ hi: aside from OLS, least squares won't result in a closed form solution when the model is non-linear so how you apply it to non-linear problems is not clear to me. unless by least squares you mean "minimizing the squares of the error terms " ? $\endgroup$
    – mlofton
    Jun 30, 2020 at 16:26
  • $\begingroup$ I just mean linear least squares fitting, that's all. $\endgroup$
    – teeeeee
    Jun 30, 2020 at 16:29
  • $\begingroup$ The standard example concerns estimating the location of a lighthouse (with a rotating light) based on random observations of where the light hits the coastline. No matter how many observations you make, the OLS estimate never improves over the estimate based on a single observation, whereas MLE converges to the true value. See stats.stackexchange.com/questions/36027 and other threads on the Cauchy distribution. $\endgroup$
    – whuber
    Jun 30, 2020 at 17:26
  • $\begingroup$ I think this answer stats.stackexchange.com/a/317696/290032 is the closest thing I've seen that helps to highlight the difference. As you can see, the answer isn't very fleshed out. I would like to see a simple distrubution of dummy data if possible. Thank you! $\endgroup$
    – teeeeee
    Jun 30, 2020 at 22:57
  • $\begingroup$ I think that example is interesting but I also think it's not conveying the general idea. Any time the true underlying model ( the model describing the response $y$ ) is NOT LINEAR, least squares is not appropriate. The more difficult part is specifying what the appropriate model is. Take a model where the response $y$ is a simple logistic function of x. That model, fitted by least squares, won't do as well as the model fitted using the logistic function. So, that's an example where least squares will do worse. $\endgroup$
    – mlofton
    Jun 30, 2020 at 23:13


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.