5
$\begingroup$

I am trying to understand maximum likelihood estimation and the role it plays in the context of machine learning.

I think MLE basically says: Find such parameters so that the model best describes the data you have.

This sounds like a no-brainer to me. I can't think of any other parameters one could possibly be looking for. Clearly, I am missing something. Can MLE be put in contrast with any other method of estimating model parameters? Can one choose not to use MLE when doing machine learning?

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ Perhaps this can be helpful, too. Also, MLE plays a central role in statistics but I would not say the same about machine learning. Actually, I think MLE's role in machine learning is minimal. There it is substituted by a loss function which takes the central role. $\endgroup$
    – Richard Hardy
    Jun 22, 2018 at 11:57
  • 2
    $\begingroup$ The reduction of MLE to "Find such parameters so that the model best describes the data you have" is not wrong, but may be misleading. The point is rather that it is a procedure (!) that yields the "best" parameter values in light of the data. That includes two things: first, that you have parameters at all, and second, that your parametrization might be wrong. So try to figure out methods, where either can happen. $\endgroup$
    – cherub
    Jun 22, 2018 at 11:58
  • 1
    $\begingroup$ MLE works well because most often probability density distributions are smooth with most of the weight of the distribution around the peak. For comparison (something you do not normally get) if you would have a theoretic error distribution that is the mixture of a small weight but high peak and a large weight but low peak, then most often your experimental values will fall in the low peak and a 'wrong' (biased and lot's of variance) MLE estimate occurs. $\endgroup$
    – Sextus Empiricus
    Jun 22, 2018 at 12:14
  • 4
    $\begingroup$ MLE pays no attention to why one is making an estimate or what the consequences of estimation errors might be (the "loss"). Since these things matter in many situations, the only reasons MLE has ever successfully been justified in these circumstances are indirect ones that show how in special cases--typically for arbitrarily large datasets--that MLE approximately minimizes the expected loss. $\endgroup$
    – whuber ♦
    Jun 22, 2018 at 13:39
  • 4
    $\begingroup$ Possible duplicate of Maximum Likelihood Estimation (MLE) in layman terms $\endgroup$
    – kjetil b halvorsen ♦
    Jun 22, 2018 at 14:56

1 Answer 1

Reset to default
5
$\begingroup$

Let's think about your characterization of what MLE is trying to do:

"Find such parameters so that the model best describes the data you have."

So, I feel that plugging in the best unbiased estimates of those parameters "best describes" the data. Could you blame me? But that's not MLE.

Using MLE for parameter estimation is not always the obvious thing to do. It may have poor finite sample size properties in your context, it may break down, or it may not even exist! And then there's the situation where you have prior information about your parameters.

Gauss in the 1800s used reasoning similar to maximum likelihood to derive the normal distribution (bottom of p. 103), a century before the concept was formalized by R.A. Fisher. It's likely that many other people used it some shape or form, and maybe you would have thought of it too, if you lived back then.

On the other hand, maximum likelihood kind of hacks the probability distribution in a way that might not have been obvious. The arguments of a probability distribution are values that the random variables can take, but maximum likelihood fixes those at sort of (but not completely) arbitrary values and then views that probability distribution as a function of totally different arguments, all to get an estimator. In all humility, I don't know that I would have thought of it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.