Minimization Method for GMM Estimates

Question

Which minimization method is usually used for estimating the coefficients in a GMM setting? I.e. in the following: \begin{align*} arg min_b \ g_T(b)'Wg_T(b) \end{align*}

Is there any dominating method or is it dependent on the specifics of your problem / the moment conditions?

Answer 1

It certainly is problem specific. E.g., for linear GMM estimators, we can find an analytical solution to the problem (using the notation from Hayashi's book, i.e., instruments $x$, regressors $z$, coefficient vector $\delta$, sample moments denoted by $S$, weighting matrix $\hat W$) $$ \widehat{\delta}(\widehat{W}):=\text{argmin}_{\tilde{\delta}}J(\tilde{\delta},\widehat{W}), $$ where $$J(\tilde{\delta},\widehat{W}):=n\cdot g_n(\tilde{\delta})'\widehat{W}g_n(\tilde{\delta}),$$ so that, unlike for nonlinear problems, there is no need for iterative methods: $J(\tilde{\delta},\widehat{W})$ can be written as $$ J(\tilde{\delta},\widehat{W})=n(s_{xy}-S_{xz}\tilde{\delta})'\widehat{W}(s_{xy}-S_{xz}\tilde{\delta}) $$ Accordingly, the first order condition for a minimum is $$ \frac{\partial}{\partial\tilde{\delta}}J(\tilde{\delta},\widehat{W})=0 $$ Multiplying out the terms in $J(\tilde{\delta},\widehat{W})$ gives $$ \frac{\partial}{\partial\tilde{\delta}}n(s_{xy}'\widehat{W}s_{xy}-2\tilde{\delta}'S_{xz}'\widehat{W}s_{xy}+\tilde{\delta}'S_{xz}'\widehat{W}S_{xz}\tilde{\delta})=0 $$ The derivative follows as \begin{eqnarray} -2S_{xz}'\widehat{W}s_{xy}+2S_{xz}'\widehat{W}S_{xz}\tilde{\delta}&=&0\notag\\ \Rightarrow\hspace{4cm}S_{xz}'\widehat{W}S_{xz}\tilde{\delta}&=&S_{xz}'\widehat{W}s_{xy}. \end{eqnarray}

Premultiplying with the (invertible for sufficiently large $n$) matrix $$(S_{xz}'\widehat{W}S_{xz})^{-1}$$ gives the solution, the GMM estimator \begin{equation} \widehat{\delta}(\widehat{W})=(S_{xz}'\widehat{W}S_{xz})^{-1}S_{xz}'\widehat{W}s_{xy} \end{equation} or equivalently, \begin{equation} \widehat{\delta}(\widehat{W})=(Z'X\widehat{W}X'Z)^{-1}Z'X\widehat{W}X'y. \end{equation}

Answer 2

It entirely depends on the form of the moment conditions! You can end up with everything from a simple, quadratic programming problem to some horrible, non-convex objective. (As a side note, a key dividing line between numerically easy and difficult to solve optimization problems is whether the problem is convex.)

As @Christoph Hanck's excellent answer describes, if the function $\mathbf{g}_T(\mathbf{b})$ follows the linear form $\mathbf{g}_T(\mathbf{b}) = A \mathbf{b}$, then minimizing $\mathbf{g}' W \mathbf{g}$ is a classic, quadratic programming (QP) optimization problem whose solution can be found by solving a linear system. Some classic examples:

• Applying GMM to the orthogonality condition $\mathbb{E}[\mathbf{x}_i \epsilon_i] = \mathbf{0}$ leads to a quadratic objective. In fact, it leads to the OLS estimator.
• In IV, applying GMM to the orthogonality condition $\mathbb{E}[\mathbf{z}_i \epsilon_i] = \mathbf{0}$ also leads to a quadratic objective. In fact, it leads to the IV estimator. (The quadratic objective is a bit more complicated though as it's a quadratic form over the inverse of a matrix. The solution ends up being found by in a sense solving two linear systems.)

More generally though, the objective need not be quadratic. For example, maximum likelihood estimation can be interpreted as GMM on the condition that the expectation of the score is zero. Depending on the likelihood function, maximum likelihood may be a simple QP problem or some non-convex horribleness.