3
$\begingroup$

We know under regularity conditions the MLE is asymptotically normal. Usually, it is said that in practice it's hard to check these assumptions. However, I wondered whether we can claim that these assumptions hold for some family of probability functions. In particular:

(provided the existence of MLE, belonging to the interior of the parameter space and iid sample)

1- is the MLE always asymptotically Normal for the Exponential family? Or it is always normal for a particular subset of this family? ( I meant one parameter, multi-parameter, or any other smaller subset of this family)

2- Generally, is there any family of distributions that we can most of the time the MLE is normal for this class?

$\endgroup$
4
$\begingroup$

I think it is important to specify what assumptions we are operating under as a baseline and what level of generality we are interested in considering. I personally enjoy working in the framework of M-Estimators as I think this level of generality is quite instructive. For this reason, let us consider the following set of assumptions. First recall the definition of an extremum estimator:

$$Q_n(\hat\theta_n)\leq\inf_{\theta\in\Theta}Q_n(\theta)+o_p(1)$$

Let us notice that MLE is a special case of this where,

$Q_n(\theta)=-n^{-1}\sum_{i}\log f(X_i, \theta)$

Now let us begin thinking about assumptions. First, let us use the assumption that $X_i$ is iid. This is actually not necessary, we can come up with asymptotic normality under non-iid data if we are willing to consider some other conditions such as weak-dependence or $\beta$-mixing. However, generally I think it is plausible to argue that data is iid given a particular context. If it is implausible in your case then we need to consider these other dependence frameworks and see if they make sense. Exchangability is another interesting framework to consider.

Now let us consider some assumptions on our criterion:

  1. $\theta_0$ is in the interior of $\Theta$
  2. The criterion, $Q_n$ is twice differentiable in some neighborhood of $\theta_0$
  3. $n^{1/2}\frac{\partial}{\partial\theta}Q_n\to^d N(0,V)$
  4. $\sup_\theta ||\frac{\partial^2}{\partial\theta\partial\theta}Q_n - H(\theta)||\to^p 0$

Finally, let us assume some things about our estimator. Firstly we need consistency,

$$\hat\theta_n \to^p \theta_0$$

Indeed, it should be clear that asymptotic normality implies consistency. Next, we need some way to tell that our criterion has found the true value, i.e. an extremum point:

$$\frac{\partial}{\partial\theta}Q_n (\hat\theta_n)=o_p(n^{-1/2})$$

I will not go through the proof of how these assumptions can be used to show asymptotic normality but it is not difficult to determine that under the assumptions above we have,

$$\sqrt{n}(\hat\theta_n - \theta_0)\to_d N(0,H_0^{-1}VH_0^{-1})$$

where $H(\theta_0)=H_0$.

Now with these preliminaries out of the way let us move on to the step of considering the assumptions that we have made. Note that we have not made any explicit assumptions on $f$ yet. However, I am assuming we have correctly specified $f$.

Let's begin by computing some of the above values that we used:

$$\frac{\partial}{\partial\theta}Q_n(\theta)=-n^{-1} \sum_i \frac{\partial}{\partial\theta}\log f(X_i, \theta)$$ $$\frac{\partial^2}{\partial\theta\partial\theta}Q_n(\theta)=-n^{-1} \sum_i \frac{\partial^2}{\partial\theta\partial\theta}\log f(X_i, \theta)$$ $$V = \mathbb{E}[\frac{\partial}{\partial\theta}\log f(X_i, \theta)\frac{\partial}{\partial\theta}\log f(X_i, \theta)]$$ $$H(\theta)=-\mathbb{E}[\frac{\partial^2}{\partial\theta\partial\theta}\log f(X_i, \theta)]$$

So assumption (2) clearly needs that $f$ is continuously differentiable. Asummption (3) will hold by the CLT for iid random variables when $\mathbb{E}[\frac{\partial}{\partial\theta}\log f(X_i, \theta))]=0$ and $\mathbb{E}[\frac{\partial}{\partial\theta}\log f(X_i, \theta))^2]<\infty$. The first part of this will hold by the first order conditions if $\theta_0$ is an interior point and that we can use the dominated convergence theorem. The second part is equivalent to verifying the Fisher information matrix equality, which reuqires that the model is correctly specified. Finally, (4) will hold as long as the second derivative satisfies a weak law of large numbers, $H_0$ is invertible, and $H(\theta)$ is continuous.

So wrapping up, it is not exactly proper to ask if this will hold for every exponential family distribution. In fact, as long as these distributions are correctly specified it should (I am not sure if there is an example of an exponential family distribution that violates some of the differentiability conditions but that would be the second thing to check). So then the only thing we really need to do is to check these primitive conditions for a specific $f$.

One quick aside: it is, of course, difficult to check if $\theta_0\in\Theta$ is an interior point but in many cases this is not contentious.

Hope this is helpful!

$\endgroup$
3
  • $\begingroup$ two questions that came to my mind are (4) how WLLN implies uniform convergence? also regarding the interior point, so mostly we may assume it is interior unless MLE is something like order statistics $\min x_i$ or $\max x_i$? $\endgroup$ – domath May 19 at 18:59
  • $\begingroup$ I misread the part about my first question in the text. 3 conditions mentioned would obviously imply that. $\endgroup$ – domath May 19 at 19:01
  • $\begingroup$ Yep! And, so for the interior question, if the MLE is on the boundary we start to run into problems. In fact, I believe the MLE cannot converge to a normal in this case. Yes, I think usually this assumption is taken as given, I am not specifically familiar with the order statistic case, but in most applications I have seen we are usually in $\mathbb{R}$. I guess some problems may occur if we need to restrict it further to say the positive reals. $\endgroup$ – Ariel May 19 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.