Simple question: does the MLE of a (finite-dimensional) parameter converge in mean square to the true value, that is, $$\mathbb E[\Vert\hat\theta_\text{MLE} - \theta\Vert^2_2]\rightarrow 0.$$ I know that, under certain conditions, the MLE is consistent, asymptotically normal, and asymptotically efficient. So I would say the answer to my question is negative.

However, consider the MLE of mean and variance of a normal distribution. It is given by $$\hat\theta_\text{MLE} = n^{-1}\sum_{i=1}^n\begin{bmatrix} X_i \\ \left(X_i - n^{-1}\sum_{i=1}^nX_i\right)^2\end{bmatrix}.$$ This MLE is asymptotically unbiased in the sense that $$\Vert\mathbb E[\hat\theta_\text{MLE} - \theta]\Vert_2\rightarrow 0,$$ and since the variance is given by $$\mathbb V[\hat\theta_\text{MLE}] = n^{-1}\begin{bmatrix}\sigma^2 & \mu_3 \\ \mu_3 & \frac{(n-1)}{n}\sigma^4\end{bmatrix},$$ it follows that $$\mathrm{trace}\big(\mathbb V[\hat\theta_\text{MLE}]\big)\rightarrow0.$$ Since $$\mathbb E[\Vert\hat\theta_\text{MLE} - \theta\Vert^2_2] = \mathrm{trace}\big(\mathbb V[\hat\theta_\text{MLE}]\big) + \Vert\mathbb E[\hat\theta_\text{MLE} - \theta]\Vert_2^2$$ the MLE converges in mean square. So does it indeed hold that the MLE converges in mean squre sense? If so, what are the conditions to establish mean square? And why is this not stated in textbooks?!

  • $\begingroup$ My guess is that there must be some counterexample, this cannot be true in full generality! $\endgroup$ – kjetil b halvorsen Feb 19 at 12:51
  • $\begingroup$ that's my guess too. However, I couldn't find one. I suppose the counter example "pathological" and thus not easy to come up with one. My problem is that, if we insist that common regularity conditions hold (e.g. compact parameter domain, $\mathcal C^p$-ness of likelihood function, etc.) to ensure the above mentioned properties, it is very hard -- at least for me -- to construct such an example $\endgroup$ – lmaosome Feb 19 at 13:15
  • $\begingroup$ Maybe you could get larger interest for your post if you edited to focus on finding such a counterexample? $\endgroup$ – kjetil b halvorsen Feb 19 at 13:20
  • 1
    $\begingroup$ I will start a bounty and hope that this will attract more attention. It may help already if someone can name sufficient conditions for mean square convergence $\endgroup$ – lmaosome Feb 19 at 15:04
  • $\begingroup$ The Neyman-Scott problem is such a counterexample where we have an inconsistent MLE stat.berkeley.edu/~census/neyscpar.pdf. You need to state some conditions for the problem to be interesting! $\endgroup$ – AdamO Feb 25 at 17:22

I think this paper answers your question! I will briefly describe the main result of the paper.

The paper suggests a family of distributions that are 'in between' the uniform and triangular distribution. That is, at $\theta = 1$ the distribution is the triangular distribution and for $\theta = 0$ the distribution is uniform. For in between values the distribution is something else.

The pdf is $$f(x \mid \theta) = \frac{(1-\theta)}{\delta(\theta)}\left(1 - \frac{|x-\theta|}{\delta(\theta)}\right)\times\mathbb{I}(x \in A) + \frac{\theta}{2}\mathbb{I}(x \in [-1,1])$$

Where $\mathbb{I}(\cdot)$ is the usual indicator function, A is the interval $[\theta - \delta(\theta), \theta + \delta(\theta)]$ and $\delta(\theta)$ is a function with the following properties

  • $\delta(\theta)$ is continuous and decreasing in $\theta$
  • $\delta(0) = 1$
  • $0<\delta(\theta)<1-\theta$.

The main result:

Let $\hat{\theta}_n$ be the MLE for $\theta$ at a sample size of $n$. If $\delta(\theta) \to 0$ sufficiently fast as $n\to \infty$ then $\hat{\theta}_n \to 1$ with probability $1$, for any true $\theta \in [0,1]$.

Therefore, the MLE cannot converge in mean square to the true value!

An example of a function $\delta(\theta)$ which causes the MLE to ''break down'' is $$\delta(\theta) = \frac{\exp(-(1-\theta)^{-2})}{1-\theta} $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.