I am estimating a non-linear GMM model. In both Stata and R, you need to specify the moment equations and the instruments, but there is no need need to provide analytical derivatives for the estimator to work, in which case these are computed numerically.

For instance, the Stata documentation (page 24) says:

By default, gmm calculates derivatives numerically, and the method used produces accurate results for the vast majority of applications. However, if you refit the same model repeatedly or else have the derivatives available, then gmm will run more quickly if you supply it with analytic derivatives.

Furthermore, regarding numerical calculation of derivatives, page 58 says:

This procedure results in accurate derivatives but can be slow if your model has many instruments or parameters.

The terminology in the documentation of the R package gmm has a stronger suggestion (regarding the specification of element gradv, the analytical derivatives):

[gradv] A function of the form $G(\theta, x)$ which returns a $q\times k$ matrix of derivatives of $\bar{g}(\theta)$ with respect to $\theta$. By default, the numerical algorithm numericDeriv is used. It is of course strongly suggested to provide this function when it is possible. This gradiant is used compute the asymptotic covariance matrix of $\hat \theta$. If "g" is a formula, the gradiant is not required (see the details below).

My intuition is that analytical derivatives are always better, because they are exact, reducing "searching" around, and thus time. But are they unambiguously better? The Stata documentation does not suggest so, whereas the R documentation seems to imply it.

  • 2
    $\begingroup$ It's hard to ever declare something "unambiguously better", but I think you get as close as possible to that in your own question when you say that "analytical derivatives are always better, because they are exact, reducing "searching" around, and thus time". Greater precision in less time seems pretty hard to beat! What other criteria might there be that would contradict that? $\endgroup$ – hamedbh Jul 17 '18 at 9:00

I can't speak about your model specifically, but in general, no.

This is part of the reason why analytical (i.e. symbolic) derivatives are not used for large models in machine learning. (Instead, algorithmic or automatic differentiation is used).

The main reason is that symbolic derivatives for large models (unless hand-optimized by a human... or sometimes because of hand-optimization by a human) tend to be unwieldly and inefficient (e.g. repeats of the same sub-expression within the larger expression). Using Mathematica or Maple for a little while will quickly show you why. They can also be numerically unstable, if one is not careful to avoid catastrophic cancellation, for example.

On the other hand, numerical derivatives have problems with efficiency as the number of parameters increases, since one has to do several function evaluations per parameter to accurately get a numerical derivative. (though one can partly evade this by taking random direction perturbations instead I suppose). As the number of parameters increases, these will be too costly. You also have to choose a step-size. But if the number of parameters is small, it should be ok. One can imagine expressions that can be quickly evaluated twice (to get a finite difference), but their analytic derivative cannot be, meaning one should just take the numerical approximation.

So really it can depend on many factors: function complexity and number of parameters (which can change independently of course), quality/availability of analytic expressions, etc...

  • $\begingroup$ Thanks. Do you have some references e.g. papers that provide simulation-based or analytical proof of this? $\endgroup$ – luchonacho Jul 18 '18 at 7:57
  • $\begingroup$ @luchonacho Hm, I think the automatic differentiation literature discusses it a bit, like this paper. $\endgroup$ – user3658307 Jul 18 '18 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.