Minimization Method for GMM Estimates

Which minimization method is usually used for estimating the coefficients in a GMM setting? I.e. in the following: \begin{align*} arg min_b \ g_T(b)'Wg_T(b) \end{align*}

Is there any dominating method or is it dependent on the specifics of your problem / the moment conditions?

• It depends on the moment conditions. – Matthew Gunn Jan 17 '18 at 16:06

It certainly is problem specific. E.g., for linear GMM estimators, we can find an analytical solution to the problem (using the notation from Hayashi's book, i.e., instruments $x$, regressors $z$, coefficient vector $\delta$, sample moments denoted by $S$, weighting matrix $\hat W$) $$\widehat{\delta}(\widehat{W}):=\text{argmin}_{\tilde{\delta}}J(\tilde{\delta},\widehat{W}),$$ where $$J(\tilde{\delta},\widehat{W}):=n\cdot g_n(\tilde{\delta})'\widehat{W}g_n(\tilde{\delta}),$$ so that, unlike for nonlinear problems, there is no need for iterative methods: $J(\tilde{\delta},\widehat{W})$ can be written as $$J(\tilde{\delta},\widehat{W})=n(s_{xy}-S_{xz}\tilde{\delta})'\widehat{W}(s_{xy}-S_{xz}\tilde{\delta})$$ Accordingly, the first order condition for a minimum is $$\frac{\partial}{\partial\tilde{\delta}}J(\tilde{\delta},\widehat{W})=0$$ Multiplying out the terms in $J(\tilde{\delta},\widehat{W})$ gives $$\frac{\partial}{\partial\tilde{\delta}}n(s_{xy}'\widehat{W}s_{xy}-2\tilde{\delta}'S_{xz}'\widehat{W}s_{xy}+\tilde{\delta}'S_{xz}'\widehat{W}S_{xz}\tilde{\delta})=0$$ The derivative follows as \begin{eqnarray} -2S_{xz}'\widehat{W}s_{xy}+2S_{xz}'\widehat{W}S_{xz}\tilde{\delta}&=&0\notag\\ \Rightarrow\hspace{4cm}S_{xz}'\widehat{W}S_{xz}\tilde{\delta}&=&S_{xz}'\widehat{W}s_{xy}. \end{eqnarray}

Premultiplying with the (invertible for sufficiently large $n$) matrix $$(S_{xz}'\widehat{W}S_{xz})^{-1}$$ gives the solution, the GMM estimator $$\widehat{\delta}(\widehat{W})=(S_{xz}'\widehat{W}S_{xz})^{-1}S_{xz}'\widehat{W}s_{xy}$$ or equivalently, $$\widehat{\delta}(\widehat{W})=(Z'X\widehat{W}X'Z)^{-1}Z'X\widehat{W}X'y.$$

• Thanks. But what about problems without an analytical solution, i.e. ones that require a numerical method? Which methods are typically used? – tstudio Jan 17 '18 at 9:42
• (+1) My only disagreement is that $\widehat{\delta}(\widehat{W})=(Z'X\widehat{W}X'Z)^{-1}Z'X\widehat{W}X'y$ is also computed using numerical methods! $\delta$ is the solution to a linear system of the form $A \boldsymbol{\delta} = \mathbf{b}$. For example, R ultimately uses the LINPACK library to form the QR decomposition of $A$ and solves with forward and back substitution. Any numerical computing uses numerical methods. – Matthew Gunn Jan 17 '18 at 15:24
• That is correct, I removed the term referring to the numerical approaches. – Christoph Hanck Jan 17 '18 at 15:28

It entirely depends on the form of the moment conditions! You can end up with everything from a simple, quadratic programming problem to some horrible, non-convex objective. (As a side note, a key dividing line between numerically easy and difficult to solve optimization problems is whether the problem is convex.)

As @Christoph Hanck's excellent answer describes, if the function $\mathbf{g}_T(\mathbf{b})$ follows the linear form $\mathbf{g}_T(\mathbf{b}) = A \mathbf{b}$, then minimizing $\mathbf{g}' W \mathbf{g}$ is a classic, quadratic programming (QP) optimization problem whose solution can be found by solving a linear system. Some classic examples:

• Applying GMM to the orthogonality condition $\mathbb{E}[\mathbf{x}_i \epsilon_i] = \mathbf{0}$ leads to a quadratic objective. In fact, it leads to the OLS estimator.
• In IV, applying GMM to the orthogonality condition $\mathbb{E}[\mathbf{z}_i \epsilon_i] = \mathbf{0}$ also leads to a quadratic objective. In fact, it leads to the IV estimator. (The quadratic objective is a bit more complicated though as it's a quadratic form over the inverse of a matrix. The solution ends up being found by in a sense solving two linear systems.)

More generally though, the objective need not be quadratic. For example, maximum likelihood estimation can be interpreted as GMM on the condition that the expectation of the score is zero. Depending on the likelihood function, maximum likelihood may be a simple QP problem or some non-convex horribleness.