Maximum likelihood vs generalized method of moments

Question

I am trying to understand how maximum likelihood (MLE) and generalized method of moments (GMM) are related to each other. In particular, I often see people saying that MLE can be written in terms of the GMM or some minimum-distance estimators. I am not sure how this is true in general.

Suppose my parameter of interest is $\theta \in \Theta \subseteq \mathbb{R}^k$, and my log-likelihood is given by $L(\theta) = \frac{1}{n} \sum^n_{i=1} f(x_i; \theta)$. Then, the first-order conditions for MLE is given by the system of $k$ equations

$$ \nabla_\theta L(\theta) = 0_{k}. $$ Equivalently, this is to solve $$ \frac{\partial L(\theta)}{\partial \theta_i} = 0 \quad \text{ for all $i = 1, \ldots, k$}. $$

From the Wikipedia page, it suggests that MLE can be written as a form of GMM using the moment conditions formed by my first-order conditions. In my notation, the moment condition is

$$ E[g(\theta)] := E[\nabla_\theta F(\theta)] = 0_k. $$

The GMM objective function is given by

$$ \min_{\theta \in \Theta} g(\theta)'W_n g(\theta), $$ for some weighting matrix $W_n$. I think one way to pick $W_n$ here is to set it as the identity matrix.

Then, the first-order condition of the GMM objective is

$$ \nabla_{\theta} g(\theta) W_n g(\theta) = 0_k. $$

Substituting back the definition of $g$ above, we have

$$ \nabla_{\theta\theta} L(\theta) W_n \nabla_{\theta} L(\theta) = 0_k. $$

My question is, how is the GMM estimator related to the MLE estimator? By looking at the first-order conditions of the two estimators, I don't think they are the same unless we impose more conditions. For instance, the second derivative appeared in the GMM first-order condition, whereas only the first derivative appeared in the MLE first-order condition.

Any thoughts are very much appreciated.

Answer 1

To expand on what I wrote in the comment, suppose we have data $\{X_i\}_{i=1}^n$ which have some distribution $F$ whose density $f_{\theta}(x)$ is known up to some parameters $\theta \in \Theta \subset \mathbb{R}^k.$ Assume that the "true" parameter is some fixed but unknown $\theta_0 \in \Theta$.

Suppose further that you know some function $g:\mathbb{R} \times \mathbb{R}^k \to \mathbb{R}^p$ of your data and unknown parameters such that $$ \begin{equation} \mathbb{E}g(X_i, \theta) = \int_{\mathbb{R}}g(x,\theta)f_{\theta_0}(x)dx = 0 \tag{1} \end{equation} $$ if and only if $\theta = \theta_0$. This is the point-identification assumption. It's an assumption because while we have a set of $p$ equations in $k$ unknowns, we don't know the equations themselves, since we need $\theta_0$ to compute the integral above. Thus we find the sample analogue of the equations, given by

$$ Q_n(\theta) := n^{-1}\sum_i g(X_i,\theta) = 0. \tag{2}$$

The identification assumption does not imply that equation (2) has a solution; it only lets us assume that equation (1) does. Indeed, what we have here is $p$ equations in $k$ unknowns; for $p < k$ neither (1) nor (2) have unique solutions, since that is an underdetermined system, so our assumption is actually invalid. For $p = k$, we are likely to have unique solutions to both systems, (in the case of linear systems we almost always have solutions, in a technical sense). For $p > k$, we assume that (1) has a unique solution, but (2) may not have a solution at all since it is an overdetermined system. This is where the quadratic form of the moments come in; we can recover estimates by minimizing $ \lVert Q_n(\theta)\rVert_2^2$ which is equivalent to making $Q_n$ as close to zero as possible. Note you can also try to minimize this quadratic form when $p = k$, but if equation (2) has solution $\hat{\theta}$, then $\lVert Q_n(\hat{\theta}) \rVert_2^2 = 0$ as well. This is exactly what is going on with your MLE example, where $g(X_i, \theta) = \nabla \log f(X_i, \theta) $.