Distribution of Maximum Likelihood Estimator

Question

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(\theta)$ be the maximum likelihood function for the distribution $f(x;\theta)$

Then after taking sample of size n

$$L(\theta)=f(x_1;\theta)\cdot f(x_2\theta)...f(x_n;\theta)$$

And we want to find $\theta_{max}$ such that $L(\theta)$ is maximized and $\theta_{max}$ is our estimate (once a sample has actually been selected)

Since $\theta_{max}$ maximizes $L(\theta)$ it also maximizes $ln(L(\theta))$

where

$$ln(L(\theta))=ln(f(x_1;\theta))+ln(f(x_2;\theta))...+ln(f(x_n;\theta))$$

Taking the derivative with respect to $\theta$

$$\frac{f'(x_1;\theta)}{f(x_1;\theta)}+\frac{f'(x_2;\theta)}{f(x_2;\theta)}...+\frac{f'(x_n;\theta)}{f(x_n;\theta)}$$

$\theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

Answer 1

MLE requires $$\frac{\partial \ln L(\theta)}{\partial \theta} = \sum_{i=1}^n \frac{ f'(x_i;\theta)}{f(x_i;\theta)},$$ where $f'(x_i;\theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;\theta)=\frac{ f'(x;\theta)}{f(x;\theta)}.$ Then $\{g(x_i;\theta)\}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;\theta)=0$. If $Eg(x_1;\theta)g(x_1;\theta)'<\infty$, CLT implies, $$\sqrt{n}(\bar{g}_n(\theta)-Eg(x_1;\theta))=\sqrt{n}\bar{g}_n(\theta) \rightarrow_D N(0,E(g(x;\theta)g(x;\theta)'),$$ where $\bar{g}_n(\theta)=\frac{1}{n} \sum_{i=1}^n g(x_i;\theta).$ The ML estimator solves the equation $$\bar{g}_n(\theta)=0.$$ It follows that the ML estimator is given by $$\hat{\theta}=\bar{g}_n^{-1}(0).$$ So long as the set of discontinuity points of $\bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $\bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $\theta$.