18
$\begingroup$

The impression that I got, based on several papers, books and articles that I've read, is that the recommended way of fitting a probability distribution on a set of data is by using maximum likelihood estimation (MLE). However, as a physicist, a more intuitive way is to just fit the pdf of the model to the empirical pdf of the data using least squares. Why then is MLE better than least squares in fitting probability distributions? Could someone please point me to a scientific paper/book that answers this question?

My hunch is because MLE does not assume a noise model and the "noise" in the empirical pdf is heteroscedastic and not normal.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

One useful way of thinking about this is to note that there are cases when least squares and the MLE are the same eg estimating the parameters where the random element has a normal distribution. So in fact, rather than (as you speculate) that the MLE does not assume a noise model, what is going on is that it does assume there is random noise, but takes a more sophisticated view of how that is shaped rather than assuming it has a normal distribution.

Any text book on statistical inference will deal with the nice properties of MLEs with regard to efficiency and consistency (but not necessarily bias). MLEs also have the nice property of being asymptotically normal themselves under a reasonable set of conditions.

Improve this answer
$\endgroup$
2
  • $\begingroup$ what I mean by "does not assume a random noise model" is that it does not assume the noise has some definite distribution e.g., normal. Could you point out a book that discusses parameter estimation by fitting PDF using least squares? The books that I found discuss MLE (and sometimes, method of moments) only. $\endgroup$
    – Christian Alis
    Mar 28, 2012 at 3:38
  • $\begingroup$ To fit MLE you still need to assume a definite distribution, but you have a wider choice than just normal. Just to pick the first book at hand that discusses the two, I have Garthwaite, Jolliffe and Jones Statistical Inference (a pretty standard second year uni text book) which discusses least squares as well as method of moments and method of minimum Chi square as alternatives to MLEs. $\endgroup$
    – Peter Ellis
    Mar 28, 2012 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.