1
$\begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(\theta)$ be the maximum likelihood function for the distribution $f(x;\theta)$

Then after taking sample of size n

$$L(\theta)=f(x_1;\theta)\cdot f(x_2\theta)...f(x_n;\theta)$$

And we want to find $\theta_{max}$ such that $L(\theta)$ is maximized and $\theta_{max}$ is our estimate (once a sample has actually been selected)

Since $\theta_{max}$ maximizes $L(\theta)$ it also maximizes $ln(L(\theta))$

where

$$ln(L(\theta))=ln(f(x_1;\theta))+ln(f(x_2;\theta))...+ln(f(x_n;\theta))$$

Taking the derivative with respect to $\theta$

$$\frac{f'(x_1;\theta)}{f(x_1;\theta)}+\frac{f'(x_2;\theta)}{f(x_2;\theta)}...+\frac{f'(x_n;\theta)}{f(x_n;\theta)}$$

$\theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

$\endgroup$
4
$\begingroup$

MLE requires $$\frac{\partial \ln L(\theta)}{\partial \theta} = \sum_{i=1}^n \frac{ f'(x_i;\theta)}{f(x_i;\theta)},$$ where $f'(x_i;\theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;\theta)=\frac{ f'(x;\theta)}{f(x;\theta)}.$ Then $\{g(x_i;\theta)\}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;\theta)=0$. If $Eg(x_1;\theta)g(x_1;\theta)'<\infty$, CLT implies, $$\sqrt{n}(\bar{g}_n(\theta)-Eg(x_1;\theta))=\sqrt{n}\bar{g}_n(\theta) \rightarrow_D N(0,E(g(x;\theta)g(x;\theta)'),$$ where $\bar{g}_n(\theta)=\frac{1}{n} \sum_{i=1}^n g(x_i;\theta).$ The ML estimator solves the equation $$\bar{g}_n(\theta)=0.$$ It follows that the ML estimator is given by $$\hat{\theta}=\bar{g}_n^{-1}(0).$$ So long as the set of discontinuity points of $\bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $\bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $\theta$.

$\endgroup$
  • $\begingroup$ $\frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that $\endgroup$ – Colin Hicks Mar 15 at 1:23
  • $\begingroup$ You're welcome :) $\endgroup$ – dlnB Mar 15 at 1:26

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.