Asymptotic normality of MLE

Question

We know under regularity conditions the MLE is asymptotically normal. Usually, it is said that in practice it's hard to check these assumptions. However, I wondered whether we can claim that these assumptions hold for some family of probability functions. In particular:

(provided the existence of MLE, belonging to the interior of the parameter space and iid sample)

1- is the MLE always asymptotically Normal for the Exponential family? Or it is always normal for a particular subset of this family? ( I meant one parameter, multi-parameter, or any other smaller subset of this family)

2- Generally, is there any family of distributions that we can most of the time the MLE is normal for this class?

Answer 1

I think it is important to specify what assumptions we are operating under as a baseline and what level of generality we are interested in considering. I personally enjoy working in the framework of M-Estimators as I think this level of generality is quite instructive. For this reason, let us consider the following set of assumptions. First recall the definition of an extremum estimator:

$$Q_n(\hat\theta_n)\leq\inf_{\theta\in\Theta}Q_n(\theta)+o_p(1)$$

Let us notice that MLE is a special case of this where,

$Q_n(\theta)=-n^{-1}\sum_{i}\log f(X_i, \theta)$

Now let us begin thinking about assumptions. First, let us use the assumption that $X_i$ is iid. This is actually not necessary, we can come up with asymptotic normality under non-iid data if we are willing to consider some other conditions such as weak-dependence or $\beta$-mixing. However, generally I think it is plausible to argue that data is iid given a particular context. If it is implausible in your case then we need to consider these other dependence frameworks and see if they make sense. Exchangability is another interesting framework to consider.

Now let us consider some assumptions on our criterion:

1. $\theta_0$ is in the interior of $\Theta$
2. The criterion, $Q_n$ is twice differentiable in some neighborhood of $\theta_0$
3. $n^{1/2}\frac{\partial}{\partial\theta}Q_n\to^d N(0,V)$
4. $\sup_\theta ||\frac{\partial^2}{\partial\theta\partial\theta}Q_n - H(\theta)||\to^p 0$

Finally, let us assume some things about our estimator. Firstly we need consistency,

$$\hat\theta_n \to^p \theta_0$$

Indeed, it should be clear that asymptotic normality implies consistency. Next, we need some way to tell that our criterion has found the true value, i.e. an extremum point:

$$\frac{\partial}{\partial\theta}Q_n (\hat\theta_n)=o_p(n^{-1/2})$$

I will not go through the proof of how these assumptions can be used to show asymptotic normality but it is not difficult to determine that under the assumptions above we have,

$$\sqrt{n}(\hat\theta_n - \theta_0)\to_d N(0,H_0^{-1}VH_0^{-1})$$

where $H(\theta_0)=H_0$.

Now with these preliminaries out of the way let us move on to the step of considering the assumptions that we have made. Note that we have not made any explicit assumptions on $f$ yet. However, I am assuming we have correctly specified $f$.

Let's begin by computing some of the above values that we used:

$$\frac{\partial}{\partial\theta}Q_n(\theta)=-n^{-1} \sum_i \frac{\partial}{\partial\theta}\log f(X_i, \theta)$$ $$\frac{\partial^2}{\partial\theta\partial\theta}Q_n(\theta)=-n^{-1} \sum_i \frac{\partial^2}{\partial\theta\partial\theta}\log f(X_i, \theta)$$ $$V = \mathbb{E}[\frac{\partial}{\partial\theta}\log f(X_i, \theta)\frac{\partial}{\partial\theta}\log f(X_i, \theta)]$$ $$H(\theta)=-\mathbb{E}[\frac{\partial^2}{\partial\theta\partial\theta}\log f(X_i, \theta)]$$

So assumption (2) clearly needs that $f$ is continuously differentiable. Asummption (3) will hold by the CLT for iid random variables when $\mathbb{E}[\frac{\partial}{\partial\theta}\log f(X_i, \theta))]=0$ and $\mathbb{E}[\frac{\partial}{\partial\theta}\log f(X_i, \theta))^2]<\infty$. The first part of this will hold by the first order conditions if $\theta_0$ is an interior point and that we can use the dominated convergence theorem. The second part is equivalent to verifying the Fisher information matrix equality, which reuqires that the model is correctly specified. Finally, (4) will hold as long as the second derivative satisfies a weak law of large numbers, $H_0$ is invertible, and $H(\theta)$ is continuous.

So wrapping up, it is not exactly proper to ask if this will hold for every exponential family distribution. In fact, as long as these distributions are correctly specified it should (I am not sure if there is an example of an exponential family distribution that violates some of the differentiability conditions but that would be the second thing to check). So then the only thing we really need to do is to check these primitive conditions for a specific $f$.

One quick aside: it is, of course, difficult to check if $\theta_0\in\Theta$ is an interior point but in many cases this is not contentious.

Hope this is helpful!

Answer 2

is the MLE always asymptotically Normal for the Exponential family? Or it is always normal for a particular subset of this family? ( I meant one parameter, multi-parameter, or any other smaller subset of this family)

Indeed, in exponential families, MLEs exist (with probability tending to 1 as $n \rightarrow \infty$) and are asymptotically normally distributed. More precisely, suppose that $\{X_n\}_{n = 1}^\infty$ are conditionally IID given $\Theta = \theta$ with nondegenerate exponential family distribution whose density with respect to a measure $\nu$ is \begin{equation} f_{\theta}(x)\ =\ c(\theta) \exp(\theta^\top x) \end{equation} Suppose that the natural parameter space $\Omega$ is an open subset of $R^k$, and let $\hat{\Theta}_n$ be the MLE of $\Theta$ based on $X_1, \dots, X_n$ if it exists. Then $\lim_{n\rightarrow \infty} P_\theta(\hat{\Theta}_n \text{ exists}) = 1$ and under $P_\theta$ \begin{equation} \sqrt{n}(\hat{\Theta}_n - \theta) \Rightarrow N(0,\ \mathcal{I}(\theta)^{-1}) \end{equation} where $\mathcal{I}(\theta)^{-1}$ is the Fisher information matrix. For a proof of this claim, look at the proof of Theorem 7.57 of Schervish's book Theory of Statistics. The proof is relatively simple, particularly when compared to the very technical proof of the MLE's asymptotic normality in a general setting.