# Why does MLE tend to normal distribution

We have $$X_1,\dots, X_n$$ are iid (the distribution can be of any type, e.g. Bernoulli (p), normal ($$\mu, \sigma^2$$), Poisson ($$\lambda$$).

If we use MLE $$\hat \theta$$ to estimate any parameter $$\theta$$ of the distribution,, then, as is said in Casella Section 10.4.1, (any continuous function of $$\hat \theta$$) $$h(\hat \theta)\to \mathrm{n}(h(\theta), v)$$, $$\quad$$(1)

where $$v$$ is $$\mathrm{Var}( h(\hat \theta))$$ as $$n\to\infty$$, which equals $$\frac{[h'(\theta)]^2}{\frac{\partial^2}{-\partial \theta^2} \log L(\theta|\mathbf{X})}|_{\theta=\hat \theta}$$. $$\quad$$(2)

My question is why, if $$X_i$$'s are iid and $$\hat \theta$$ is MLE, $$h(\hat \theta)$$ tends to normal distribution as $$n\to\infty$$? I can understand that if $$\hat \theta=\bar X$$, then it seems to be just CLT. But why it is so when $$\hat \theta$$ is not average of sample.

The reason given in the text is that MLE is asymptotically efficient (theorem 10.1.12), but would anyone give an easier explanation? BTW, can we say CLT is just a special case of theorem 10.1.12?

It's said according to Slutsky's Theorem (i.e. if $$X_n,Y_n,Z_n$$ are sequence of estimator tending to a random variable) $$X$$ of certain distribution, constant a, b, $$Y_nX_n+Z_n\to aX+b$$), $$\frac{h(\hat \theta)-h(\theta)}{\sqrt{\hat{\mathrm{Var}}( h(\hat \theta))}}\to \mathrm{n}(0,1)$$. Why?

My thought is that both $$h(\hat \theta)$$ and $$\sqrt{\hat{\mathrm{Var}}( h(\hat \theta))}$$ are random variables (since they are estimators, and estimators are functions of/dependent on the sample $$X_i$$'s, which are random variables), tending to a random var $$\mathrm{n}(h(\theta), v)$$ (according to (1)) and a random var (2) (for (2) is function of random var $$\mathbf{x}$$ and random var $$\hat \theta$$; so we can't use Slutsky's Theorem directly.

But, say $$\sigma^2$$ is the variance of $$\theta$$, and we have $$\frac{h(\hat \theta)-h(\theta)}{\sqrt{\hat{\mathrm{Var}}( h(\hat \theta))}}=\frac{h(\hat \theta)-h(\theta)}{\sigma}\frac{\sigma}{\sqrt{\hat{\mathrm{Var}}( h(\hat \theta))}}$$, the first factor tends to $$\mathrm{n}(0,1)$$, the second seems to tend to 1 (Why? It is still a random variable and should tend to a distribution instead of a number), and therefore we get the result.

My question is as bolded.

From what's said above we have confidence interval $$h(\hat \theta)-z_{\alpha/2} \sqrt{\hat{\mathrm{Var}}( h(\hat \theta))} < h(\theta) < h(\hat \theta)+z_{\alpha/2} \sqrt{\hat{\mathrm{Var}}( h(\hat \theta))}$$, $$\quad$$ (5)

that is, according to Definition 9.1.4, 9.1.5 of confidence coefficient, $$1-\alpha=\inf P\left(L(\mathbf{X}) < h(\theta) < U(\mathbf{X})\right),$$ $$\quad$$ (4)

where $$L(\mathbf{X})=h(\hat \theta)-z_{\alpha/2} \sqrt{\hat{\mathrm{Var}}( h(\hat \theta))}, U(\mathbf{X}) = h(\hat \theta)+z_{\alpha/2} \sqrt{\hat{\mathrm{Var}}( h(\hat \theta))}$$. $$\quad$$ (5)

(We can verify, as we have done in the above section, that the two 'bounds' are random variables and functions of $$\mathbf{X}=(X_1,\dots, X_n)$$.) And infimum is of the sequence of the set/sequence of the above probability for all $$n\in\mathbb{N}_+$$.

To make this point (and that it's not about probability of $$h(\theta)$$ but about that of $$\hat \theta, \hat{\mathrm{Var}}( h(\hat \theta))$$) more explicit we can write (4) as $$\inf P\left(\mathbf{X}^{-1}(L^{-1}(-\infty, h(\theta))), \mathbf{X}^{-1}(U^{-1}(h(\theta), \infty)))\right).$$

How can we directly proceed from the definition of confidence interval to (5), i.e. given the definition and (5), how to prove (4), that is, $$\inf P\left(\mathbf{X}^{-1}(L^{-1}(-\infty, h(\theta))), \mathbf{X}^{-1}(U^{-1}(h(\theta), \infty)))\right)=1-\alpha$$?

(Updated: I think the key step is we form one static $$T(\mathbf{X})$$ from the two bounds $$L(\mathbf{X}), U(\mathbf{X})$$ and we can calculate its distribution, possibly it’s normal distribution—-if no we can average several such statics and from CLT we know this ‘mean’ has normal distribution. (This seems to be what we do in Casella Exercise 10.8). From this we can easily tell the probability of the parameter to be estimated is in $$(L(\mathbf{X}), U(\mathbf{X}))$$. The key here is that $$h(\theta)$$ and its estimator is ‘symmetric’ in our static $$T(\mathbf{X})$$.)

My another question is that why we define confidence coefficient (for interval estimator) this way, in particular, why we need to include 'infimum' in this definition? I ask this because the confidence level $$\alpha$$ is defined as $$1-\alpha= P\left( -z_{\alpha/2} < X < z_{\alpha/2} \right)$$; here $$X$$, not the upper and lower bounds, is random variable; and we don't use 'infimum'.

Updated:

The question in the first section can be partly answered by checking the proof of theorem 10.1.12, $$h(\hat \theta)$$ has normal distribution because $$\sqrt{n}(\hat \theta-\theta)$$ is (with Taylor expansion) proportional to $$l'(\theta|\mathbf{x})=\frac1n\sum_iW_i$$ (Exercise 10.8 of Casella) where $$W_i=\frac d {d\theta} (\log f(x_i|\theta))=\frac{\frac d {d\theta} f(x_i|\theta)}{{f(x_i|\theta)}}$$ and is iid, has, according to CLT, normal distribution.

I'm wondering how to prove that $$W_i$$ is iid.

• Nov 4, 2020 at 20:52
• @kjetil b halvorsen They seem useful. Nov 4, 2020 at 23:36