Proof that ML Estimator is asymptotically Normal

Question

I'm trying to prove that the Maximum Likelihood Estimator is Asymptotically Normal distributed.

I'm stuck in the lasts steps. Here's what I've done:

I do the Taylor's expansion of, that's the mean of the score function: $$\frac{1}{n}\sum \frac{\partial \log f(x_i, \theta)}{\partial \theta}$$ The Taylor's expansion around the true, unknown, value $\theta_0$ is:

$$ \left.\frac{1}{n}\sum \frac{\partial \log f(x_i, \theta)}{\partial \theta}\right\vert_{\theta_0}+ \left.\frac{1}{n}\sum \frac{\partial^2 \log f(\underline{x}, \theta)}{\partial \theta^2}\right\vert_{\theta_0}(\theta-\theta_0) +R/n $$

We know that the mean is an approximation of the expected value thanks to Weak Law of Large Numbers. The first one goes to 0 and second goes to a $-I_n(\theta)$ and the third goes to 0 for assumptions on the form of the remainder.

Now my problem is that I was told to use the ML estimation $\hat\theta$ and do again the Taylor's expansion, but I didn't get all the steps

I know only that in the end we get this:

$$ (\hat{\theta}-\theta_0)=\left[\frac{1}{\sqrt{n}}\sum \frac{\partial^2 \log f(\underline{x}, \theta)}{\partial \theta^2}\right]^{-1}\left[\frac{1}{\sqrt{n}}\sum \frac{\partial \log f(x_i, \theta)}{\partial \theta}+ R/n\right] $$

$ \hat\theta-\theta_0 \sim N(0,I^{-1}(\theta_0)) $

I know that we have to use the Central Limit Theorem, but I'm quite confused and I don't know how to go on. I tried to get some information, but with no results. Can someone provide me a clear explanation on why the MLE goes to Normal asymptotically?

Answer 1

The log likelihood function is $$l(\theta_0)=\sum_{i=1}^n \log(f(x_i)) \tag{1}$$ Since $\hat{\theta}$ is a solution of the maximum of log likelihood function $l(\theta_0)$ we know that $l'(\hat{\theta})=0$.

Next we do a Taylor expansion of $l'(\hat{\theta})$ around $\theta_0$

$$l'(\hat{\theta})=l'(\theta_0)+\frac{l''(\theta_0)}{1!}(\hat{\theta}-\theta_0)+\frac{l'''(\theta)}{2!}(\hat{\theta}-\theta_0)^2$$

Since $l'(\hat{\theta})=0$, we do some rearrangements here,

$$-l''(\theta_0)(\hat{\theta}-\theta_0)-\frac{l'''(\theta_0)}{2}(\hat{\theta}-\theta_0)^2=l'(\theta_0)$$ $$(\hat{\theta}-\theta_0)=\frac{l'(\theta_0)}{-l''(\theta_0)-\frac{l'''(\theta)}{2}(\hat{\theta}-\theta_0)}$$

We multiply $\sqrt{n}$ at both sides we get

$$\sqrt{n}(\hat{\theta}-\theta_0)=\frac{\frac{1}{\sqrt{n}}l'(\theta_0)}{-\frac{1}{n}l''(\theta_0)-\frac{l'''(\theta)}{2n}(\hat{\theta}-\theta_0)} \tag{2}$$

Next we need to show that $\frac{1}{\sqrt{n}} l'(\theta_0)$ has a $N(0,I(\theta_0))$ distribution.

From $(1)$ we get

$$l'(\theta_0)=\sum_{i=1}^n\frac{\partial \log(f(x_i))}{\partial \theta_0}$$

We multiply $\frac{1}{\sqrt{n}}$ at both side.

$$\frac{1}{\sqrt{n}}l'(\theta_0)=\frac{1}{\sqrt{n}}\sum_{i=1}^n\frac{\partial \log(f(x_i))}{\partial \theta_0} \tag{3}$$

Now we use CLT for the right hand side of $(3)$

We treat $\frac{\partial \log(f(x_i))}{\partial \theta_0}$ as a random variable here.

And we can show $E\left(\frac{\partial \log(f(x_i))}{\partial \theta_0}\right)=0$ by following procedures:

$$1=\int_{-\infty}^{\infty}f(x)dx$$ take derivative of both sides

$$0=\int_{-\infty}^{\infty}\frac{\partial f(x)}{\partial \theta_0}dx=\int_{-\infty}^{\infty}\frac{\partial f(x)}{\partial \theta_0 f(x)}f(x)dx=\int_{-\infty}^{\infty}\frac{\partial \log(f(x))}{\partial \theta_0}f(x)dx$$

which shows that $E\left(\frac{\partial \log(f(x_i))}{\partial \theta_0}\right)=0$

We can show the variance of $\frac{\partial \log(f(x_i))}{\partial \theta_0}$ is $I(\theta_0)$

Therefore, $$\frac{1}{\sqrt{n}}l'(\theta_0)\sim N(0,I(\theta_0))$$

We also can show that $-\frac{1}{n}l''(\theta_0)=I(\theta_0)$. I will not do detailed derivations here.

We also ignore the $-\frac{l'''(\theta)}{2}(\hat{\theta}-\theta_0)$ part in $(2)$

Now we wrap up $(2)$

$$\sqrt{n}(\hat{\theta}-\theta_0) \sim \frac{N(0,I(\theta_0))}{I(\theta_0)}=N\left(0,\frac{1}{I(\theta_0)}\right)$$

By some rearrangements, you can see $\hat{\theta}$ also normally distributed.