When maximizing a likelihood function $L(\psi)$, the gradient-based optimization procedure is generally $$ \tag{5.1} \hat{\psi}_{r+1} = \hat{\psi}_{r} + \left| I^{*}(\hat{\psi}_{r}) \right|^{-1} D \log L(\hat{\psi}_{r}) $$ where $D \log L(\hat{\psi}_{r})$ is the gradient and $I^{*}(\hat{\psi}_{r})$ is some approximation to the information matrix (for instance the Hessian in Newton-Raphson), evaluated at $\psi = \hat{\psi}_{r}$ In Harvey (1990), it says that a single iteration of this procedure gives a consistent and asymptotically efficient estimator, if $\hat{\psi}_{0}$ was the "initial consistent estimator" of $\psi$.

Here's the (abridged and highlighted) relevant quote from Harvey (1990), pp. 140-42.

(5.1) defines a fairly general iterative scheme in which a variety of expressions may be used for $I^{*}(\psi)$. The essential point is that under standard regularity conditions, $I^{*}(\psi)$ is asymptotically equivalent to the information matrix [...]. A two-step estimator is constructed by evaluating the r.h.s. of (5.1) at a consistent estimator of $\psi$. In other words only one iteration is carried out, with $\hat{\psi}$ being the initial consistent estimator of $\psi^{*}$ the two-step estimator. The attraction of this procedure is that the two-step estimator has the same asymptotic distribution as the ML estimator. [...] Estimators formed by one iteration of (5.1) are sometimes referred to as linearised maximum likelihood. [...] Iterating (5.1) further will usually result in an estimate which is closer to the ML solution, although this certainly cannot be guaranteed. However, iterating beyond the first round will not change the asymptotic distribution of the estimator even if continuing the process eventually produces the ML estimator.

The book then continues with a proof for the case of a single parameter, but says "a general proof may be constructed along similar lines."

But as a practical matter, how would this work? When and how would one have a starting value $\hat{\psi}_{0}$ that just so happens to be consistent? Or is this just a theoretical result with no practical implications?

EDIT: I dug a little deeper, and found this 1975 article by Bickel. He talks about estimating equations more generally. Here's the relevant quote, where I adjusted the notation to Harvey's above:

$$ \tag{1.1} \sum_{i=1}^{n} \phi (X_{i} - \hat{\psi}) = 0 $$ where $X_{i} = \psi + E_{1}, \ldots, X_{n} = \psi +E_{n}$ and $E_{1}, \ldots, E_{n}$ are unknown independent, identically distributed errors which have a distribution $F$ which is symmetric about 0. If $F$ has a density $f$ which is smooth and if $f$ is known, then maximum likelihood estimates if they exist satisfy (1.1) with $\phi = -f'/f$. [...] It has, however, been observed by Fisher, Neyman and others that if $F$ is known and $\phi = -f'/f$, the estimate obtained by starting with a $\sqrt{n}$ consistent estimate $\hat{\psi}_{0}$ and performing one Gauss-Newton iteration of (1.1) is asymptotically efficient even when the MLE is not and is equivalent to it when it is.

My question remains: where would this initial consistent estimate $\hat{\psi}_{0}$ come from?

  • $\begingroup$ Unable to view link. Says I have reached max number of allowed page views. Is it possible to share the relevant details in a public domain? $\endgroup$ – Aleksejs Fomins Mar 12 at 20:21
  • 1
    $\begingroup$ I added a relevant quote of the material, which should be considered fair use. I also changed the notation to match the book. $\endgroup$ – Durden Mar 12 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.