21
$\begingroup$

In general it seems like the method of moments is just matching the observed sample mean, or variance to the theoretical moments to get parameter estimates. This is often the same as MLE for exponential families, I gather.

However, it's hard to find a clear definition of the method of moments and a clear discussion of why the MLE seems to be generally favored, even though it can be trickier to find the mode of the likelihood function.

This question Is MLE more efficient than Moment method? has a quote from Prof. Donald Rubin (at Harvard) saying that everyone has known since the 40s that MLE beats MoM, but I'd be interested to know the history or reasoning for this.

$\endgroup$
2

4 Answers 4

13
+25
$\begingroup$

What is the method of moments?

There is a nice article about this on Wikipedia.

It means that you are estimating the population parameters by selecting the parameters such that for certain specific moments the population distribution has the moments that are equivalent to the observed moments in the sample.

How is it different from MLE

The maximum likelihood estimate maximizes the likelihood function. In some cases this maximum can sometimes be expressed in terms of setting the population parameters equal to the sample parameters.

E.g. when estimating the mean parameter of a distribution and employ MLE then often we end up with using $\mu = \bar{x} $. However this does not need to be always the case ( related: https://stats.stackexchange.com/a/317631/164061 although in the case of the example there, the Poisson distribution, the MLE and MoM estimate coincide, and the same is true for many others). For example the MLE solution for the estimate of $\mu $ in a log normal distribution is:

$$\mu = 1/n \sum \ln (x_i) = \overline {\ln (x)}$$

Whereas the MoM solution is solving

$$\exp \left(\mu + \frac {1}{2}\sigma^2\right) = \bar {x}$$ leading to $$\mu = \ln (\bar {x}) - \frac {1}{2} \sigma^2$$


So the MoM is a practical way to estimate the parameters, leading often to the exact same result as the MLE (since the moments of the sample often coincide with the moments of the population, e.g. a sample mean is distributed around the population mean, and up to some factor/bias, it works out very well). The MLE has a stronger theoretical foundation and for instance allows estimation of errors using the Fisher matrix (or estimates of it), and it is a much more natural approach in the case of regression problems (I haven't tried it but I guess that a MoM for solving parameters in a simple linear regression is not working easily and may give bad results. In the answer by superpronker it seems like this is done by some minimization of a function. For MLE this minimization expresses higher probability, but I wonder whether it represents such a similar thing for MoM).

$\endgroup$
5
  • 3
    $\begingroup$ @develarist Fisher wrote some things about it in the early 1920s. Starting with 'on mathematical foundations of theoretical statistics' and 'A Mathematical Examination of the Methods of Determining the Accuracy of an Observation by the Mean Error, and by the Mean Square Error'. $\endgroup$ Nov 16, 2020 at 6:12
  • $\begingroup$ @ Sextus: Great answer! I have often herd that "Method of Moments" is more robust to misspecification (i.e. assuming the wrong probability distribution function) compared to Maximum Likelihood Estimation ... but I have never seen a proof of this. Do you know if this exists? $\endgroup$
    – stats_noob
    Aug 2, 2023 at 4:14
  • 1
    $\begingroup$ @stats_noob I believe that this just depends on what moments, or more generally statistics, are used for the estimation. It needs to be evaluated on a case by case basis. For example, with a Laplace distribution, the median is the MLE and the mean is a MoM estimator. Few people would claim that the mean is more robust than the median. Finding a proof would also be difficult as 'robustness' is not well defined. $\endgroup$ Aug 2, 2023 at 8:03
  • $\begingroup$ @SextusEmpiricus: is it possible to make a specific example? For example - for a specific probability distribution... can we show that method of moments produces a more robust estimator ot the mean when compared to mle? $\endgroup$
    – stats_noob
    Aug 2, 2023 at 12:52
  • $\begingroup$ @stats_noob robustness is a very subjective term (or at least not defined in a clear/unique way, that leaves little room for changing it). Because of that it will always be possible to make some example where a MoM estimator does better than the MLE. Just pick a case where a MoM estimator and the MLE are different and figure out some definition of robust that favours the MoM estimator $\endgroup$ Aug 2, 2023 at 15:37
12
$\begingroup$

In MoM, the estimator is chosen so that some function has conditional expectation equal to zero. E.g. $E[g(y,x,\theta)] = 0$. Often the expectation is conditional on $x$. Typically, this is converted to a problem of minimizing a quadratic form in this expectations with a weight matrix.

In MLE, the estimator maximizes the log likelihood function.

In broad generalization, MLE makes stricter assumptions (the full density) and is thus typically less robust but more efficient if the assumptions are met (it achieves the Kramer Rao lower bound on asymptotic variance).

In some cases the two coincide, OLS being one notable example where the analytic solution is identical and hence the estimator behaves in the same way.

In some sense, you can think of an MLE (in almost all cases) as an MoM estimator because the estimator sets the expected value of the gradient of the log likelihood function equal to zero. In that sense, there are cases where the density is incorrect but the MLE is still consistent because the first order conditions are still satisfied. Then MLE is referred to as "quasi-ML".

$\endgroup$
2
  • 6
    $\begingroup$ Usually, with MoM ones refer to the case where the function g is some power so the expectation is a moment. This looks more like "generalized method of moments". $\endgroup$ Dec 22, 2016 at 22:54
  • 9
    $\begingroup$ OLS is a method of moments estimator (MoME). It is also a maximum likelihood estimator (MLE), but only for a special case of likelihood -- the normal one. For another distribution, OLS will not be a MLE, while it still is a MoME. $\endgroup$ Dec 23, 2016 at 7:04
6
+50
$\begingroup$

The MLE is invariant by transformation of the data $(X_1,...,X_n)$ by a strictly increasing transformation. Concretely, if $g$ is strictly increasing and $Y_i=g(X_i)$ and the $X_i$'s have density $f_{\theta}$, then the $Y_i$'s have density $$ F_\theta: y\mapsto f_\theta(g^{-1}(y)) (g^{-1})'(y). $$ Minimizing the negative log-likelihood for $(X_1,...,X_n)$ gives $\hat \theta = \text{argmin}_{\theta} -\sum_{i=1}^n \log f_\theta(X_i)$. Minimizing the negative log-likelihood for $(Y_1,...,Y_n)$ gives the same MLE because $$ -\sum_{i=1}^n \log F_\theta(Y_i) = -\sum_{i=1}^n \log f_\theta(X_i) + \log((g^{-1})'(Y_i) $$ so that the second term does not depend on $\theta$: Minimizing $-\sum_{i=1}^n \log F_\theta(Y_i)$ is the same as minimizing $-\sum_{i=1}^n \log f_\theta(X_i)$. So the MLE $\hat \theta$ is invariant under monotonic transformation.

The same cannot be said for method of moments estimators, for which monotonic transformation can lead to terrible results. Consider $X_i\sim N(\theta,1)$ for an unknown parameter $\theta>0$ so that the MLE is the average $\hat\theta=\frac1n\sum_{i=1}^nX_i$. The method of moments for $Y_1,...,Y_n$ where $Y_i=g(X_i)$ for $g(t)=t^2$ suggests to consider an estimator $\tilde\theta$ that solves the equation $$ \theta^2 + 1 = E_\theta[X^2] = \frac 1 n \sum_{i=1}^n X_i^2, $$ so that $\tilde\theta = (\frac 1 n\sum_{i=1}^n X_i^2 - 1)^{1/2}$. By the law of large numbers, $\frac 1 n\sum_{i=1}^n X_i^2\to^P \theta^2+1$ so that $\tilde \theta \to^P \theta$ by the continuous mapping theorem. But the asymptotic variance of $\tilde\theta$ can be arbitrarily bad: we have $\frac{1}{\sqrt n}\sum_{i=1}^n(X_i^2 - 1)-\theta^2 \to^d N(0, 2)$ since $Var[\chi^2_1]=2$ (variance of $X_i$ which has the chi-square distribution with 1 degree of freedom). The delta-method applied the function $s(u)=\sqrt{u}$ function then gives $$ \frac{1}{\sqrt n}(\tilde \theta - \theta) \to^d N(0, 2s'(\theta^2)^2) = N(0, 1/(2\theta)). $$ The asymptotic variance can get arbitrarily bad as $\theta$ approaches 0, while the asymptotic variance of the MLE is always 1, i.e., $n^{-1/2}(\hat\theta-\theta)=^d N(0,1)$.

Summary

The MLE enjoys optimal asymptotic variance thanks to the Cramer-Rao lower bound, and the MLE is invariant by monotonic transformation of $(X_1,...,X_n)$. On the other hand, method of moments estimates can be rendered arbitrarily bad (in terms of asymptotic variance) by certain monotonic transformation of $(X_1,...,X_n)$.

$\endgroup$
4
$\begingroup$

Sorry, I can't post comments..

MLE makes stricter assumptions (the full density) and is thus typically less robust but more efficient if the assumptions are met

Actually, at MITx "Fundamentals of Statistics," we are taught the opposite, that MoM relies on the specific equation of the moments, and if we pick up the wrong density, we do totally wrong, while MLE is more resilient, as we in all case minimise the KD divergence.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.