# Distribution of Maximum Likelihood Estimator

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $$L(\theta)$$ be the maximum likelihood function for the distribution $$f(x;\theta)$$

Then after taking sample of size n

$$L(\theta)=f(x_1;\theta)\cdot f(x_2\theta)...f(x_n;\theta)$$

And we want to find $$\theta_{max}$$ such that $$L(\theta)$$ is maximized and $$\theta_{max}$$ is our estimate (once a sample has actually been selected)

Since $$\theta_{max}$$ maximizes $$L(\theta)$$ it also maximizes $$ln(L(\theta))$$

where

$$ln(L(\theta))=ln(f(x_1;\theta))+ln(f(x_2;\theta))...+ln(f(x_n;\theta))$$

Taking the derivative with respect to $$\theta$$

$$\frac{f'(x_1;\theta)}{f(x_1;\theta)}+\frac{f'(x_2;\theta)}{f(x_2;\theta)}...+\frac{f'(x_n;\theta)}{f(x_n;\theta)}$$

$$\theta_{max}$$ would be the solution of the above when set to 0 (after selecting values for all $$x_1,x_2...x_n$$) but why is it normally distributed and how do I show that it's true for large n?

MLE requires $$\frac{\partial \ln L(\theta)}{\partial \theta} = \sum_{i=1}^n \frac{ f'(x_i;\theta)}{f(x_i;\theta)},$$ where $$f'(x_i;\theta)$$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $$g(x;\theta)=\frac{ f'(x;\theta)}{f(x;\theta)}.$$ Then $$\{g(x_i;\theta)\}_{i=1}^n$$ is a new iid sequence of random variables, with $$Eg(x_1;\theta)=0$$. If $$Eg(x_1;\theta)g(x_1;\theta)'<\infty$$, CLT implies, $$\sqrt{n}(\bar{g}_n(\theta)-Eg(x_1;\theta))=\sqrt{n}\bar{g}_n(\theta) \rightarrow_D N(0,E(g(x;\theta)g(x;\theta)'),$$ where $$\bar{g}_n(\theta)=\frac{1}{n} \sum_{i=1}^n g(x_i;\theta).$$ The ML estimator solves the equation $$\bar{g}_n(\theta)=0.$$ It follows that the ML estimator is given by $$\hat{\theta}=\bar{g}_n^{-1}(0).$$ So long as the set of discontinuity points of $$\bar{g}_n^{-1}(z)$$, i.e. the set of all values of $$z$$ such that $$\bar{g}_n^{-1}(z)$$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $$\theta$$.
• $\frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that Mar 15, 2019 at 1:23