5
$\begingroup$

I am trying to understand how maximum likelihood (MLE) and generalized method of moments (GMM) are related to each other. In particular, I often see people saying that MLE can be written in terms of the GMM or some minimum-distance estimators. I am not sure how this is true in general.

Suppose my parameter of interest is $\theta \in \Theta \subseteq \mathbb{R}^k$, and my log-likelihood is given by $L(\theta) = \frac{1}{n} \sum^n_{i=1} f(x_i; \theta)$. Then, the first-order conditions for MLE is given by the system of $k$ equations

$$ \nabla_\theta L(\theta) = 0_{k}. $$ Equivalently, this is to solve $$ \frac{\partial L(\theta)}{\partial \theta_i} = 0 \quad \text{ for all $i = 1, \ldots, k$}. $$

From the Wikipedia page, it suggests that MLE can be written as a form of GMM using the moment conditions formed by my first-order conditions. In my notation, the moment condition is

$$ E[g(\theta)] := E[\nabla_\theta F(\theta)] = 0_k. $$

The GMM objective function is given by

$$ \min_{\theta \in \Theta} g(\theta)'W_n g(\theta), $$ for some weighting matrix $W_n$. I think one way to pick $W_n$ here is to set it as the identity matrix.

Then, the first-order condition of the GMM objective is

$$ \nabla_{\theta} g(\theta) W_n g(\theta) = 0_k. $$

Substituting back the definition of $g$ above, we have

$$ \nabla_{\theta\theta} L(\theta) W_n \nabla_{\theta} L(\theta) = 0_k. $$

My question is, how is the GMM estimator related to the MLE estimator? By looking at the first-order conditions of the two estimators, I don't think they are the same unless we impose more conditions. For instance, the second derivative appeared in the GMM first-order condition, whereas only the first derivative appeared in the MLE first-order condition.

Any thoughts are very much appreciated.

$\endgroup$
1
  • $\begingroup$ The deep connection between MLE and GMM can be shown properly by looking at theory of extremum estimators in general. For now, it should be enough to note that the quadratic form which you see in the GMM objective is only needed if you have more moment conditions than parameters. Here the moment conditions are $k$ partial derivatives, and the number of parameters is $k$, so the sample analogue of $E[\nabla \log f(x_i, \theta)] = 0$ is exactly the GMM objective. In other words, you are solving $n^{-1}\sum_i \nabla \log f(x_i,\hat{\theta})] = 0$ for $\hat{\theta}$. $\endgroup$ Feb 16, 2023 at 23:18

1 Answer 1

1
$\begingroup$

To expand on what I wrote in the comment, suppose we have data $\{X_i\}_{i=1}^n$ which have some distribution $F$ whose density $f_{\theta}(x)$ is known up to some parameters $\theta \in \Theta \subset \mathbb{R}^k.$ Assume that the "true" parameter is some fixed but unknown $\theta_0 \in \Theta$.

Suppose further that you know some function $g:\mathbb{R} \times \mathbb{R}^k \to \mathbb{R}^p$ of your data and unknown parameters such that $$ \begin{equation} \mathbb{E}g(X_i, \theta) = \int_{\mathbb{R}}g(x,\theta)f_{\theta_0}(x)dx = 0 \tag{1} \end{equation} $$ if and only if $\theta = \theta_0$. This is the point-identification assumption. It's an assumption because while we have a set of $p$ equations in $k$ unknowns, we don't know the equations themselves, since we need $\theta_0$ to compute the integral above. Thus we find the sample analogue of the equations, given by

$$ Q_n(\theta) := n^{-1}\sum_i g(X_i,\theta) = 0. \tag{2}$$

The identification assumption does not imply that equation (2) has a solution; it only lets us assume that equation (1) does. Indeed, what we have here is $p$ equations in $k$ unknowns; for $p < k$ neither (1) nor (2) have unique solutions, since that is an underdetermined system, so our assumption is actually invalid. For $p = k$, we are likely to have unique solutions to both systems, (in the case of linear systems we almost always have solutions, in a technical sense). For $p > k$, we assume that (1) has a unique solution, but (2) may not have a solution at all since it is an overdetermined system. This is where the quadratic form of the moments come in; we can recover estimates by minimizing $ \lVert Q_n(\theta)\rVert_2^2$ which is equivalent to making $Q_n$ as close to zero as possible. Note you can also try to minimize this quadratic form when $p = k$, but if equation (2) has solution $\hat{\theta}$, then $\lVert Q_n(\hat{\theta}) \rVert_2^2 = 0$ as well. This is exactly what is going on with your MLE example, where $g(X_i, \theta) = \nabla \log f(X_i, \theta) $.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.