# Proof that ML Estimator is asymptotically Normal

I'm trying to proof that the Maximum Likelihood Estimator is Asymptotically Normal distribuited. I'm stuck in the lasts steps. Here's what I've done:

I do the Taylor's expansion of, that's the mean of the score function: $$\frac{1}{n}\sum \frac{\partial log f(x_i, \theta)}{\partial \theta}$$ The Taylor's expansion around the true, unknown, value $\theta_0$ is: $$\left.\frac{1}{n}\sum \frac{\partial log f(x_i, \theta)}{\partial \theta}\right\vert_{\theta_0}+ \left.\frac{1}{n}\sum \frac{\partial^2 log f(\underline{x}, \theta)}{\partial \theta^2}\right\vert_{\theta_0}(\theta-\theta_0) +R/n$$ We know that the mean is an approximatio of the expected value thanks to Weak Law of Large Numbers. The first one goes to 0 and second goes to a $-I_n(\theta)$ and the third goes to 0 for assumptions on the form of the remainder.

Now my problem is that I was told to use the ML estimation $\hat\theta$ and do again the Taylor's expansion, but I didn't get all the steps

I know only that in the end we get this: $$(\hat{\theta}-\theta_0)=\left[\frac{1}{\sqrt{n}}\sum \frac{\partial^2 log f(\underline{x}, \theta)}{\partial \theta^2}\right]^{-1}\left[\frac{1}{\sqrt{n}}\sum \frac{\partial log f(x_i, \theta)}{\partial \theta}+ R/n\right]$$

$\hat\theta-\theta_0 \sim N(0,I^{-1}(\theta_0))$

I know that we have to use the Central Limit Theorem, but I'm quite confused and I don't know how to go on. I tried to get some information, but with no results. Can someone provide me a clear explanation on why the MLE goes to Normal asymptotically? Thank you.

The log likelihood funciton is $$l(\theta_0)=\sum_{i=1}^n log(f(x_i)) \tag{1}$$ Since $\hat{\theta}$ is a solution of the maximum of log likelihood funtion $l(\theta_0)$ we know that $l'(\hat{\theta})=0$.

Next we do a Taylor expanson of $l'(\hat{\theta})$ around $\theta_0$

$$l'(\hat{\theta})=l'(\theta_0)+\frac{l''(\theta_0)}{1!}(\hat{\theta}-\theta_0)+\frac{l'''(\theta)}{2!}(\hat{\theta}-\theta_0)^2$$

Since $l'(\hat{\theta})=0$, we do some rearrangements here,

$$-l''(\theta_0)(\hat{\theta}-\theta_0)-\frac{l'''(\theta_0)}{2}(\hat{\theta}-\theta_0)^2=l'(\theta_0)$$ $$(\hat{\theta}-\theta_0)=\frac{l'(\theta_0)}{-l''(\theta_0)-\frac{l'''(\theta)}{2}(\hat{\theta}-\theta_0)}$$

We multiply $\sqrt{n}$ at both sides we get

$$\sqrt{n}(\hat{\theta}-\theta_0)=\frac{\frac{1}{\sqrt{n}}l'(\theta_0)}{-\frac{1}{n}l''(\theta_0)-\frac{l'''(\theta)}{2n}(\hat{\theta}-\theta_0)} \tag{2}$$

Next we need to show that $\frac{1}{\sqrt{n}} l'(\theta_0)$ has a $N(0,I(\theta_0))$ distribution.

From $(1)$ we get

$$l'(\theta_0)=\sum_{i=1}^n\frac{\partial log(f(x_i))}{\partial \theta_0}$$

We multiply $\frac{1}{\sqrt{n}}$ at both side.

$$\frac{1}{\sqrt{n}}l'(\theta_0)=\frac{1}{\sqrt{n}}\sum_{i=1}^n\frac{\partial log(f(x_i))}{\partial \theta_0} \tag{3}$$

Now we use CLT for the right hand side of $(3)$

We treat $\frac{\partial log(f(x_i))}{\partial \theta_0}$ as a random variable here.

And we can show $E(\frac{\partial log(f(x_i))}{\partial \theta_0})=0$ by following procedures:

$$1=\int_{-\infty}^{\infty}f(x)dx$$ take derivative of both sides

$$0=\int_{-\infty}^{\infty}\frac{\partial f(x)}{\partial \theta_0}dx=\int_{-\infty}^{\infty}\frac{\partial f(x)}{\partial \theta_0 f(x)}f(x)dx=\int_{-\infty}^{\infty}\frac{\partial log(f(x))}{\partial \theta_0}f(x)dx$$

Which shows that $E(\frac{\partial log(f(x_i))}{\partial \theta_0})=0$

We can show the variance of $\frac{\partial log(f(x_i))}{\partial \theta_0}$ is $I(\theta_0)$

Therefore, $$\frac{1}{\sqrt{n}}l'(\theta_0)\sim N(0,I(\theta_0))$$

We also can show that $-\frac{1}{n}l''(\theta_0)=I(\theta_0)$. I will not do detailed derivations here.

We also ignore the $-\frac{l'''(\theta)}{2}(\hat{\theta}-\theta_0)$ part in $(2)$

Now we wrap up $(2)$

$$\sqrt{n}(\hat{\theta}-\theta_0) \sim \frac{N(0,I(\theta_0))}{I(\theta_0)}=N(0,\frac{1}{I(\theta_0)})$$

By some rearrangements, you can see $\hat{\theta}$ also normally distributed.