7
$\begingroup$

The problem I have is in figuring out why the MLE is no longer consistent in countable parameter spaces under conditions specified below.

The set up is as follows: we are consider a parameters space $\Theta \subseteq \mathbb R$ and a set of probability distributions, $\mathcal P = \{P_\theta: \theta \in \Theta\}$.

We assume the following conditions hold:

  1. The distributions $P_\theta$ of the observations are distinct
  2. The distributions $P_\theta$ have common support.
  3. The observations are $\mathbf X_n = \{X_1,...,X_n\}$, where $X_i$ are iid with probability density $f(x_i| \theta)$ with respect to a sigma-finite measure $\mu$.

I take as given that the following theorem holds (see, for example, Theorem 6.1.1 in Hogg et al., Introduction to Mathematical Statistics (7th ed.)). Also, throughout, we denote by $\theta_0$ the "true" parameter that generated the data.


Result 1. Under conditions 1. to 3. the likelihood function $$L(\theta|\mathbf X_n) = \prod_{i=1}^n f(X_i|\theta)$$ satisfies $$P_{\theta_0}[L(\theta_0| \mathbf X_n) > L(\theta| \mathbf X_n)] \rightarrow 1\text{ as } n\rightarrow \infty$$ for any fixed $\theta\ne\theta_0$.


I want to show that under the same conditions, plus the assumption that the parameter space is finite, the estimator that maximizes the likelihood function is consistent. Further, I want to understand why this result breaks down when the parameter space is at least countably infinite.

So, here's my attempted proof for the finite case.


As $\Theta$ is finite, we may write $\Theta = \{\theta_0, \theta_1,...,\theta_m\}$, where $\theta_0$ is the true parameter. Let $$\hat\theta(\mathbf X_n) = \arg\max_{\theta\in \Theta}L(\theta| \mathbf X_n) $$ be the MLE. We want to show that there is a unique value $\hat\theta(\mathbf X_n)$ that maximizes the likelihood and that it tends to $\theta_0$ in probability as $n \rightarrow\infty$.

For each $1\le j\le m$, let $A_{jn} =\{\mathbf X_n : L(\theta_0| \mathbf X_n)>L(\theta_j | \mathbf X_n)\}$. Then, the result above shows that $P_{\theta_0}[A_{jn}] \rightarrow 1$ as $n\rightarrow\infty$ for all $j$. What remains to be shown is that $P_{\theta_0}[A_{1n}\cap A_{2n}\cap \cdots \cap A_{mn}] \rightarrow 1$ as $n\rightarrow\infty$ as well. It will suffice to show that this result holds for any $A_{jn}\cap A_{j'n}$, $j\ne j'$ (as we can repeatedly apply this result $m-1$ times). We have $$P_{\theta_0}[A_{jn}\cap A_{j'n}] = 1- P_{\theta_0}[A_{jn}^C\cup A_{j'n}^C] \ge 1 - P_{\theta_0}[A_{jn}^C] - P_{\theta_0}[A_{j'n}^C]$$ for all $n$ by the sub-additivity of probability measure. Thus, $$ P_{\theta_0}[A_{jn}\cap A_{j'n}] = 1$$ as $n\rightarrow\infty$, since $P_{\theta_0}[A_{jn}^C]$ and $P_{\theta_0}[A_{j'n}^C]$ converge to zero. Hence, $$P_{\theta_0}[A_{1n}\cap A_{2n}\cap \cdots \cap A_{mn}] \rightarrow 1.$$ In other words, the probability that the likelihood function evaluated at $\theta_0$ is simultaneously (strictly) larger than any other possible parameter value converges to one. By condition 1., however, this value must be unique with probability converging to one. Thus, by choosing the value of $\theta$ that maximizes the likelihood function---i.e., $\hat\theta(\mathbf X_n)$---we will choose the true parameter value $\theta_0$ with probability converging to one. This is equivalent to the statement that $$P_{\theta_0}[\hat\theta(\mathbf X_n) = \theta_0] \rightarrow 1$$ as $n\rightarrow\infty$. Thus, $\hat\theta(\mathbf X_n)$ is consistent.


Now, I wonder why this result breaks down when $\Theta$ is assumed to be (countably) infinite. My attempt so far is the following:

Suppose that $\Theta$ is countably infinite. Then, $$\begin{aligned} \lim_{n\rightarrow\infty} P_{\theta_0}\left[\bigcap_{j=1}^\infty A_{jn}\right] &= 1 - \lim_{n\rightarrow\infty} P_{\theta_0}\left[\bigcup_{j=1}^\infty A_{jn}^C \right] = 1 - \lim_{n\rightarrow\infty} P_{\theta_0}\left[\lim_{m\rightarrow\infty} \bigcup_{j=1}^m A_{jn}^C \right] \\ &= 1 - \lim_{n\rightarrow\infty} \lim_{m\rightarrow\infty} P_{\theta_0}\left[ \bigcup_{j=1}^m A_{jn}^C \right] \end{aligned}$$ where the last step follows from $B_{mn} = \bigcup_{j=1}^m A_{jn}^C$ being an increasing sequence in $m$ and the continuity property of probability measure. If we could interchange the two limits, then $\lim_{n\rightarrow\infty}P_{\theta_0}\left[\bigcap_{j=1}^\infty A_{jn}\right] =1$ as for each fixed $m$, $\lim_{n\rightarrow\infty}P_{\theta_0}\left[\bigcup_{j=1}^m A_{jn}^C\right] = 0$ . Yet, for any finite $n$, we can find $\epsilon(n)>0$ such that $$\lim_{m\rightarrow\infty} P_{\theta_0}[B_{mn}] > \epsilon(n).$$

If it would be possible to find an $\epsilon$ that does not depend on $n$ or if we can show that $\inf_n \epsilon(n) > 0$, I think this would prove the result. I'm, however, not sure whether we can show this.

Am I am on the right track? Any suggestion would be helpful. Thanks!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.