25
$\begingroup$

Maximum likelihood estimation often results into biased estimators (e.g., its estimate for the sample variance is biased for the Gaussian distribution).

What then makes it so popular? Why exactly is it used so much? Also, what in particular makes it better than the alternative approach -- method of moments?

Also, I noticed that for the Gaussian, a simple scaling of the MLE estimator makes it unbiased. Why is this scaling not a standard procedure? I mean -- Why is it that after MLE computation, it is not routine to find the necessary scaling to make the estimator unbiased? The standard practice seems to be the plain computation of the MLE estimates, except of course for the well known Gaussian case where the scaling factor is well known.

$\endgroup$
  • 11
    $\begingroup$ There are many, many alternatives to ML, not just the method of moments--which also tends to produce biased estimators, by the way. What you might want to ask instead is "why would anybody want to use an unbiased estimator?" A good way to start researching this issue is a search on bias-variance tradeoff. $\endgroup$ – whuber Nov 22 '15 at 16:28
  • 6
    $\begingroup$ As whuber pointed out, there is no intrinsic superiority in being unbiased. $\endgroup$ – Xi'an Nov 22 '15 at 16:38
  • 4
    $\begingroup$ I think @whuber means "why would anybody want to use a biased estimator?" It doesn't take much work to convince someone that an unbiased estimator may be a reasonable one. $\endgroup$ – Cliff AB Nov 22 '15 at 17:31
  • 5
    $\begingroup$ See en.wikipedia.org/wiki/… for an example where the only unbiased estimator is certainly not one you'd want to use. $\endgroup$ – Scortchi - Reinstate Monica Nov 22 '15 at 19:24
  • 4
    $\begingroup$ @Cliff I intended to ask the question in its more provocative, potentially more mysterious form. Lurking behind this is the idea that there are many ways to evaluate the quality of an estimator and many of them have nothing to do with bias. From that point of view, it is most natural to ask why someone would propose an unbiased estimator. See glen_b's answer for more from this point of view. $\endgroup$ – whuber Nov 23 '15 at 16:44
18
$\begingroup$

Unbiasedness isn't necessarily especially important on its own.

Aside a very limited set of circumstances, most useful estimators are biased, however they're obtained.

If two estimators have the same variance, one can readily mount an argument for preferring an unbiased one to a biased one, but that's an unusual situation to be in (that is, you may reasonably prefer unbiasedness, ceteris paribus -- but those pesky ceteris are almost never paribus).

More typically, if you want unbiasedness you'll be adding some variance to get it, and then the question would be why would you do that?

Bias is how far the expected value of my estimator will be too high on average (with negative bias indicating too low).

When I'm considering a small sample estimator, I don't really care about that. I'm usually more interested in how far wrong my estimator will be in this instance - my typical distance from right... something like a root-mean-square error or a mean absolute error would make more sense.

So if you like low variance and low bias, asking for say a minimum mean square error estimator would make sense; these are very rarely unbiased.

Bias and unbiasedness is a useful notion to be aware of, but it's not an especially useful property to seek unless you're only comparing estimators with the same variance.

ML estimators tend to be low-variance; they're usually not minimum MSE, but they often have lower MSE than than modifying them to be unbiased (when you can do it at all) would give you.

As an example, consider estimating variance when sampling from a normal distribution $\hat{\sigma}^2_\text{MMSE} = \frac{S^2}{n+1}, \hat{\sigma}^2_\text{MLE} = \frac{S^2}{n}, \hat{\sigma}^2_\text{Unb} = \frac{S^2}{n-1}$ (indeed the MMSE for the variance always has a larger denominator than $n-1$).

$\endgroup$
  • 1
    $\begingroup$ +1. Is there any intuition for (or perhaps some theory behind) your second-before last paragraph? Why do ML estimators tend to be low-variance? Why do they often have lower MSE than the unbiased estimator? Also, I am amazed to see the expression for MMSE estimator of variance; somehow I have never encountered it before. Why is it so rarely used? And does it have anything to do with shrinkage? It seems that it is "shrunk" from unbiased towards zero, but I am confused by that as I am used to thinking about shrinkage only in the multivariate context (along the lines of James-Stein). $\endgroup$ – amoeba says Reinstate Monica Nov 25 '15 at 1:16
  • 1
    $\begingroup$ @amoeba MLEs are generally functions of sufficient statistics, and at least asymptotically minimum variance unbiased, so you expect them to be low variance in large samples, typically achieving the CRLB in the limit; this is often reflected in smaller samples. $\:$ MMSE estimators are generally shrunk toward zero because that reduces variance (and hence a small amount of bias toward 0 introduced by a small shrinkage will typically reduce MSE). $\endgroup$ – Glen_b -Reinstate Monica Nov 25 '15 at 4:16
  • $\begingroup$ @Glen_b, great answer (I keep coming back to it). Would you have an explanation or a reference for $\hat{\sigma}^2_\text{MMSE} = \frac{S^2}{n+1}$ being the minimum MSE estimator? $\endgroup$ – Richard Hardy Apr 26 '18 at 8:56
  • $\begingroup$ Also, does that imply the ML estimator of variance is not a minimum-variance estimator? Otherwise the minimum MSE estimator would be some weighted average (with positive weights) of the MLE and the unbiased estimator, but now it is outside that range. I could ask this as a separate question if you think it makes sense. $\endgroup$ – Richard Hardy Apr 26 '18 at 9:12
  • 1
    $\begingroup$ Found a whole derivation in a Wikipedia article on MSE, I guess that explain all of it. $\endgroup$ – Richard Hardy Apr 26 '18 at 13:02
16
$\begingroup$

MLE yields the most likely value of the model parameters, given the model and the data at hand -- which is a pretty attractive concept. Why would you choose parameter values that make the data observed less probable when you can choose the values that make the data observed the most probable across any set of values? Would you wish to sacrifice this feature for unbiasedness? I do not say the answer is always clear, but the motivation for MLE is pretty strong and intuitive.

Also, MLE may be more widely applicable than method of moments, as far as I know. MLE seems more natural in cases of latent variables; for example, a moving average (MA) model or a generalized autoregressive conditional heteroskedasticity (GARCH) model can be directly estimated by MLE (by directly I mean it is enough to specify a likelihood function and submit it to an optimization routine) -- but not by method of moments (although indirect solutions utilizing the method of moments may exist).

$\endgroup$
12
$\begingroup$

Actually, the scaling of the maximum likelihood estimates in order to obtain unbiased estimates is a standard procedure in many estimation problems. The reason for that is that the mle is a function of the sufficient statistics and so by the Rao-Blackwell theorem if you can find an unbiased estimator based on sufficient statistics, then you have a Minimum Variance Unbiased Estimator.

I know that your question is more general than that but what I mean to emphasize is that key concepts are intimately related to the likelihood and estimates based on it. These estimates might not be unbiased in finite samples but they are asymptotically so and moreover they are asymptotically efficient, i.e. they attain the Cramer-Rao bound of variance for unbiased estimators, which might not always be the case for the MOM estimators.

$\endgroup$
11
$\begingroup$

To answer your question of why the MLE is so popular, consider that although it can be biased, it is consistent under standard conditions. In addition, it is asymptotically efficient, so at least for large samples, the MLE is likely to do as well or better as any other estimator you may cook up. Finally, the MLE is found by a simple recipe; take the likelihood function and maximize it. In some cases, that recipe may be hard to follow, but for most problems, it is not. Plus, once you have this estimate, we can derive the asymptotic standard errors right away using Fisher's information. Without using the Fisher's information, it is often really hard to derive the error bounds.

This is why MLE estimation is very often the go to estimator (unless you're a Bayesian); it's simple to implement and likely to be just as good if not better than anything else you need to do more work to cook up.

$\endgroup$
  • 1
    $\begingroup$ Can you please elaborate as to how it compares to the method of moments, as this seems to be an important part of the OP? $\endgroup$ – Antoni Parellada Nov 22 '15 at 17:41
  • 1
    $\begingroup$ as pointed out by whuber, the MOM estimators are also biased, so there's not a "unbiased-ness" advantage to the MOM estimators. Also, when the MOM and MLE estimators disagree, the MLE tends to have lower MSE. But this answer is really about why MLE's tend to be the default, rather than a direct comparison to other methods. $\endgroup$ – Cliff AB Nov 22 '15 at 18:19
  • 2
    $\begingroup$ @AntoniParellada There is an interesting thread in comparing MLE and MoM, stats.stackexchange.com/q/80380/28746 $\endgroup$ – Alecos Papadopoulos Nov 22 '15 at 18:54
3
$\begingroup$

I'd add that sometimes (often) we use an MLE estimator because that's what we got, even if in an ideal world it wouldn't be what we want. (I often think of statistics as being like engineering, where we use what we got, not what we want.) In many cases it's easy to define and solve for the MLE, and then get a value using an iterative approach. Whereas for a given parameter in a given situation there may be a better estimator (for some value of "better"), but finding it may require being very clever; and when you're done being clever, you still only have the better estimator for that one particular problem.

$\endgroup$
  • 1
    $\begingroup$ Out of curiosity, what's an example of what (in the ideal world) you would want? $\endgroup$ – Glen_b -Reinstate Monica Nov 24 '15 at 23:11
  • 2
    $\begingroup$ @Glen_b: Dunno. Unbiased, lowest variance, easy to compute in closed form? When you first learn the estimators for least-squares regression, life seems simpler than it turns out to be. $\endgroup$ – eac2222 Nov 30 '15 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.