Suppose instead of maximizing likelihood I maximize some other function g. Like likelihood, this function decomposes over x's (ie, g({x1,x2})=g({x1})g({x2}), and "maximum-g" estimator is consistent. How do I compute asymptotic variance of this estimator?

Update 8/24/10: Percy Liang goes through derivation of asymptotic variance in a similar setting in An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators.

Update 9/14/10: Most useful theorem seems to be from Van der Vaart's "Asymptotic Statistics"

Under some general regularity conditions, distribution of this estimator approaches normal centered around its expected value $\theta_0$ with covariance matrix

$$\frac{\ddot{g}(\theta_0)^{-1} E[\dot{g}\dot{g}^T] \ddot{g}(\theta_0)^{-1}}{n}$$

Where $\ddot{g}$ is a matrix of second derivatives, $\dot{g}$ is gradient, $n$ is number of samples


2 Answers 2


I think the standard solution goes as follows. I'll just do the scalar case, the multi parameter case is similar. Your objective function is $g_N(p,X_1,\dots,X_N)$ where $p$ is the parameter you want to estimate and $X_1,\dots,X_N$ are the observed random variables. For notational simplicity I will just write the objective function as $g(p)$ from now on.

We need some assumptions. Firstly I'll assume that you have already shown that the maximiser of $g$ is a consistent estimator (this actually tends to be the hardest part!). So, if the `true' value of the parameter is $p_0$ and the estimator is $$ \hat{p} = \arg\max_{p} g(p) $$ then we have that $\hat{p} \rightarrow p_0$ almost surely as $N \rightarrow \infty$. Our second assumption is that $g$ is twice differentiable in a neighbourhood about $p_0$ (you can sometimes get away without this assumption, but the solution becomes more problem dependent). In view of strong consistency we can and will assume that $\hat{p}$ is inside this neighbourhood.

Denote by $g'$ and $g''$ the first and second derivatives of $g$ with respect to $p$. Then $$ g'(p_0) - g'(\hat{p}) = (p_0 - \hat{p})g''(\bar{p})$$ where $\bar{p}$ lies between $\hat{p}$ and $p_0$. Now because $\hat{p}$ maximises $g$ we have $g'(\hat{p}) = 0$ so $$(p_0 - \hat{p}) = \frac{g'(p_0)}{g''(\bar{p})}$$ and because $\hat{p} \rightarrow p_0$ almost surely then $\bar{p} \rightarrow p_0$ almost surely so $g''(\bar{p}) \rightarrow g''(p_0)$ almost surely and $$(p_0 - \hat{p}) \rightarrow \frac{g'(p_0)}{g''(p_0)}$$ almost surely. So, in order to describe the distribution of $p_0 - \hat{p}$, i.e. the estimators central limit theorem, you need to find the distribution of $\frac{g'(p_0)}{g''(p_0)}$. This now becomes problem dependent.

  • $\begingroup$ I should point out that I have left out any scale factors (e.g. $\sqrt{N}$ is common in these problems) but they can be added without trouble. What you choose for a scale factor will depend on the distribution of $g'(p_0)$. $\endgroup$ Sep 9, 2010 at 12:31
  • $\begingroup$ Yup, and page 51 of van der Vaart's "Asymptotic Statistics" gives the distribution of that quantity as Gaussian with covariance matrix $$E[g'^2]/E[g'']^2$$ I don't see why consistency is that important here...Taylor expanding around the biased estimate p* instead of p0 seems to produce Gaussian with the same variance by this argument $\endgroup$ Sep 10, 2010 at 18:43
  • $\begingroup$ Sure, a biased estimate converging to $p^*$ is going to have a similar central limit theorem, i.e. $p^* - \hat{p}$ must then be assumed to converge. This is simply replacing $p_0$ by $p^*$. $\endgroup$ Sep 10, 2010 at 23:37

The consistency and asymptotic normality of the maximum likelihood estimator is demonstrated using some regularity conditions on the likelihood function. The wiki link on consistency and asymptotic normality has the conditions necessary to prove these properties. The conditions at the wiki may be stronger than what you need as they are used to prove asymptotic normality whereas you simply want to compute the variance of the estimator.

I am guessing that if your function satisfies the same conditions then the proof will carry over to your function as well. If not then we need to know one or both of the following: (a) the specific condition that $g(.)$ does not satisfy from the list at the wiki and (b) the specifics of $g(.)$ to give a better answer to your question.

  • $\begingroup$ Hm...following the derivations under asymptotic normality link I'm a bit confused...what's the difference between I and H? $\endgroup$ Aug 19, 2010 at 20:14
  • $\begingroup$ There is no difference. See the definition of fisher information i.e., I where it is re-written as H. I guess the variance expression can be simplified on the wiki as $H^{-1} I$ is an identity matrix. $\endgroup$
    – user28
    Aug 19, 2010 at 20:32
  • $\begingroup$ Hm....it says that H becomes I when the model is correctly specified, this makes me wonder what's the difference when it is not $\endgroup$ Aug 19, 2010 at 20:50
  • $\begingroup$ Where does it say that? I am a bit unsure what the model correctness has got to do with the relationship between H and I. $\endgroup$
    – user28
    Aug 19, 2010 at 22:36
  • $\begingroup$ At the bottom of the asymptotic normality derivation from your link. It says "Finally, the information equality guarantees that when the model is correctly specified, matrix H will be equal to the Fisher information I, so that the variance expression simplifies to just I^−1" $\endgroup$ Aug 19, 2010 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.