How to compute efficiency?

Question

Suppose instead of maximizing likelihood I maximize some other function g. Like likelihood, this function decomposes over x's (ie, g({x1,x2})=g({x1})g({x2}), and "maximum-g" estimator is consistent. How do I compute asymptotic variance of this estimator?

Update 8/24/10: Percy Liang goes through derivation of asymptotic variance in a similar setting in An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators.

Update 9/14/10: Most useful theorem seems to be from Van der Vaart's "Asymptotic Statistics"

Under some general regularity conditions, distribution of this estimator approaches normal centered around its expected value $\theta_0$ with covariance matrix

$$\frac{\ddot{g}(\theta_0)^{-1} E[\dot{g}\dot{g}^T] \ddot{g}(\theta_0)^{-1}}{n}$$

Where $\ddot{g}$ is a matrix of second derivatives, $\dot{g}$ is gradient, $n$ is number of samples

Answer 1

I think the standard solution goes as follows. I'll just do the scalar case, the multi parameter case is similar. Your objective function is $g_N(p,X_1,\dots,X_N)$ where $p$ is the parameter you want to estimate and $X_1,\dots,X_N$ are the observed random variables. For notational simplicity I will just write the objective function as $g(p)$ from now on.

We need some assumptions. Firstly I'll assume that you have already shown that the maximiser of $g$ is a consistent estimator (this actually tends to be the hardest part!). So, if the `true' value of the parameter is $p_0$ and the estimator is $$ \hat{p} = \arg\max_{p} g(p) $$ then we have that $\hat{p} \rightarrow p_0$ almost surely as $N \rightarrow \infty$. Our second assumption is that $g$ is twice differentiable in a neighbourhood about $p_0$ (you can sometimes get away without this assumption, but the solution becomes more problem dependent). In view of strong consistency we can and will assume that $\hat{p}$ is inside this neighbourhood.

Denote by $g'$ and $g''$ the first and second derivatives of $g$ with respect to $p$. Then $$ g'(p_0) - g'(\hat{p}) = (p_0 - \hat{p})g''(\bar{p})$$ where $\bar{p}$ lies between $\hat{p}$ and $p_0$. Now because $\hat{p}$ maximises $g$ we have $g'(\hat{p}) = 0$ so $$(p_0 - \hat{p}) = \frac{g'(p_0)}{g''(\bar{p})}$$ and because $\hat{p} \rightarrow p_0$ almost surely then $\bar{p} \rightarrow p_0$ almost surely so $g''(\bar{p}) \rightarrow g''(p_0)$ almost surely and $$(p_0 - \hat{p}) \rightarrow \frac{g'(p_0)}{g''(p_0)}$$ almost surely. So, in order to describe the distribution of $p_0 - \hat{p}$, i.e. the estimators central limit theorem, you need to find the distribution of $\frac{g'(p_0)}{g''(p_0)}$. This now becomes problem dependent.

Answer 2

The consistency and asymptotic normality of the maximum likelihood estimator is demonstrated using some regularity conditions on the likelihood function. The wiki link on consistency and asymptotic normality has the conditions necessary to prove these properties. The conditions at the wiki may be stronger than what you need as they are used to prove asymptotic normality whereas you simply want to compute the variance of the estimator.

I am guessing that if your function satisfies the same conditions then the proof will carry over to your function as well. If not then we need to know one or both of the following: (a) the specific condition that $g(.)$ does not satisfy from the list at the wiki and (b) the specifics of $g(.)$ to give a better answer to your question.