Proof of consistency of Maximum Likelihood Estimator

Question

I would appreciate some help comprehending a logical step in the proof below about the consistency of MLE. It comes directly from Hogg, McKean, Craig, Introduction to Mathematical Statistics, 6th edition, Chapter 6, page 317.

So here goes:

Assume that $\hat{\theta_n}$ solves the estimating equation $\frac{\partial l(\theta)}{\partial \theta}=0$. Also assume the following regularity conditions:

1. the parameters identify the pdfs
2. the pdfs have common support for all $\theta$
3. The point $\theta_0$ is the true parameter and is an interior point in $\Omega$

Then $\hat{\theta_n} \xrightarrow{P} \theta_0$

Proof

Let $\mathbf{X}=(x_1,x_2, \ldots, {x_n})$, the vector of observations. Since $\theta_0$ is an interior point in $\Omega$ , $(\theta_0 -a, \theta_0 +a) \subset \Omega $ for some $a >0$. Define $S_n$ to be the event

$$S_n= \{ \mathbf {X} : l(\theta_0 ; \mathbf{X}) > l(\theta_0 -a ; \mathbf{X}) \} \cap \{ \mathbf{X}: l(\theta_0; \mathbf{X}) > l( \theta_0 +a ;\mathbf{X}) \} $$

And this is where i have a problem. How can we evaluate the log-likelihood function at $\theta=\theta_0\pm a $ ? we only know that $(\theta_0 -a, \theta_0 +a) \subset \Omega $, we have not assumed that $\theta_0 -a\in\Omega$ or that $\theta_0 +a\in\Omega$. What if $\Omega=(\theta_0 -a, \theta_0 +a)$? Then the set $S_n$ is always empty(because $l(\theta_0 -a ; \mathbf{X})$ and $l(\theta_0 +a ; \mathbf{X})$ are undefined. ) and the proof breaks down for trivial reasons.

So that is my question. How is it justified to assume that $l(\theta_0 +a ; \mathbf{X})$ and $l(\theta_0 -a ; \mathbf{X})$ are defined?

Of course, the proof is not complete at this point but if I have this clarified, I can take it from there. Thank you in advance.

edit:

until now, I have seen this same proof in two books, namely Lehman Casella(2nd edition) page 447 and Hogg and Craig(6th edition) page 317. and in both books, the authors did not assume that $\theta-a$ and $\theta+a$ are inside the parameter space. And so, I am of the opinion that assuming $\theta-a\in\Omega$ and $\theta+a\in\Omega$ is unnecessary. What I do not understand is why these assumptions are unnecessary.

Answer 1

It looks like the author intends for the set $(\theta_0-a, \theta_0+a)$ to be on the interior of $\Omega$. Strictly speaking, what the proof should have done is to choose a value $a>0$ such that:

$$(\theta_0-a, \theta_0+a) \subseteq \text{int }\Omega,$$

which then implies $\theta \pm a \in \Omega$. Bear in mind here that we only need to have the relevant equations hold for some $a>0$, and this can be guaranteed by taking $a$ to be sufficiently small to ensure that $(\theta_0-a, \theta_0+a)$ lies in the interior of the parameter space. I think the author of the proof was just a bit sloppy here in stating that particular step.

Answer 2

I am not sure that I understand your concern, but sounds like it is that possibly $\theta-a$ or $\theta+a$ might be on the boundary of the parameter space, with one being either the maximum or the minimum of $\Omega$.

I agree that you need to assume both $\theta-a$ and $\theta+a$ are interior points in the parameter space. It is just a fact that even when you get a proof from a textbook sometimes small details (and sometimes large ones) are left out. Usually when a proof says for 'some $a$' it means there is at least one $a>0$ that works, and for $a$ to work you also need it to have $\theta-a$ and $\theta+a$ in the parameter space. In fact the above proof doesn't even say that $\theta+a$ or $\theta-a$ has to be anywhere inside or even close to being inside the parameter space. Also, although I am not sure how the rest of the proof proceeds, but I am guessing at some point it takes a limit as $a$ goes to $0$? That means it does not really matter how close the original value of $a$ is to 0.

Answer 3

I am just a student but here an other prove that works if you all ready know by hypothesis that the MLE estimator is unbiased at least when the size sample goes to infinity.

1. $\theta_n $ is the MLE estimator of an i.i.d sample $\left \{ X_1=x_1, X_2=x_2,\ldots,X_n=x_n \right \}$ of size $n$ and $\theta_0$ the parameter to estimate.
   First let note that the sequence $(\theta_n)_n$ is a sequence of r.v. while $\theta_0$ is a parameter (a "constant").
   So it is asking to us to prove: $$\forall \epsilon>0 \Rightarrow \lim_{n \to \infty }\mathbb{P}(|\theta_n - \theta_0| \geq \epsilon)=0$$

2. In order to do that we will use the Markov Inequality with $r=2$ (the condition are verify by question hypothesis):
   $$\forall \epsilon>0 \Rightarrow \mathbb{P}(|\theta_n - \theta_0| \geq \epsilon) \leq \frac{E(|\theta_n-\theta_0|^2)}{\epsilon^2}=\frac{E((\theta_n-\theta_0)^2)}{\epsilon^2}=\frac{E(\theta_n^2+\theta_0^2-2\theta_n \theta_0)}{\epsilon^2}=\frac{E(\theta_n^2)+\theta_0^2-2\theta_0E(\theta_n)}{\epsilon^2}$$

3. As we know that $\theta_n$ is unbiased when $n$ goes to infinity it means that: $$\lim_{n \to \infty }E(\theta_n)-\theta_0=0 \Rightarrow \lim_{n \to \infty }E(\theta_n) = \theta_0 = E(\theta_0)$$

Hence $E(\theta_n)$ converges in law to $E(\theta_0)=\theta_0$. So because $f(x)=x^2$ is a continuous function:$$\lim_{n \to \infty }E(\theta_n^2)=E(\theta_0^2) = \theta_0^2$$

Q.E.D.

I hope this is correct.