# Consistency of the extremum estimators

I am trying to understand intuitively the assumptions mentioned in the following theorem:

Theorem. If there is a function $Q_0(\theta)$ s.t. (i) $Q_0(\theta)$ is uniquely maximized at $\theta_0$; (ii) $\Theta$ is compact; (iii) $Q_0(\theta)$ is continuous; (iv) $\hat{Q}_n(\theta)$ converges uniformly in probability to $Q_0(\theta)$, then $\hat{\theta} \rightarrow \theta_0$ in probability.

I understand that the last assumption says that $\hat{Q}_n(\theta)$ is between $Q_0(\theta) - \varepsilon$ and $Q_0(\theta) + \varepsilon$. for some $\varepsilon$. Thus, given assumption (i) we can find $\theta_L < \theta_0 < \theta_U$ s.t. $\hat{\theta} \in (\theta_L, \theta_U)$. Does assumption (i) guarantee that $|\theta_0| \neq \infty$? Why intuitively assumptions (ii) and (iii) are needed?

• Compactness implies closed on a metric space, so (ii) is what guarantees that $|\theta_0| \neq \infty$. – jbowman Aug 8 '14 at 18:37
• @jbowman What does go wrong if $|\theta_0| = \infty$? – Kolibris Aug 8 '14 at 18:46
• This is the consistency theorem for Extremum Estimators in general (of which M-estimators are a subset). It would be useful to modify the title of the question accordingly. – Alecos Papadopoulos Aug 8 '14 at 19:24
• You can't converge in probability to infinity, it's always infinitely far away from any real number, so your estimator isn't going to be consistent. – jbowman Aug 8 '14 at 20:06
• @jbowman makes perfect sense. Do you have any idea how I can interpret closed parameter space and continuity intuitively? – Kolibris Aug 8 '14 at 20:20

The OP asked for "intuitive" explanations of the assumptions, but what's "intuitive" varies greatly from person to person. So I will provide a view of these assumptions as a "reasonable framework" in order to examine consistency of the extremum estimator, and not as a set of assumptions that are designed to be sufficient for consistency. In my eyes at least, this is an interesting way to gain "intuition" on them.

The extremum estimator $\hat \theta_n$ is defined as (I am indexing the estimator also with $n$ for a reason),

$$\hat \theta_n : Q_n(\hat \theta_n) = \max_{\theta \in \Theta}Q_n(\theta) \tag{1}$$

And how do we know that $\hat \theta_n$ exists? How do we know that $Q_n(\cdot)$ has a maximum in $\Theta$?

We don't: In order to guarantee it we have to assume that a) $Q_n(\cdot)$ is continuous in $\theta$ and b) that $\Theta$ is compact. (since compactness of the set from which the argument of the function takes its values is equivalent to the existence of a maximum of a continuous function in this set). This is the first and foremost reason why we assume that the parameter space is compact (together with continuity of $Q_n(\cdot)$: to guarantee the existence of the extremum estimator -but it will serve us in other aspects too.

Moreover, since "the limit function $Q_0(\theta)$ is uniquely maximized at $\theta_0$" appears as an assumption, then, together with compactness of $\Theta$ imply the continuity of $Q_0(\cdot)$ (so assumption (iii) in the question is in reality redundant, although it is usually stated as an additional assumption).

So why do we make the assumption that "the limit function $Q_0(\cdot)$ is uniquely maximized at $\theta_0$?" I would offer the following intuition: In order to "target" the unknown parameter of interest, it must possess some distinctive characteristic, on which to base the targeting. So we start by finding a limit function $Q_0(\cdot)$ such that $\theta_0$ has a special role in relation to it, compared to other values in the parameter space. One such special role is for $\theta_0$ to be the unique $\text{argmax}$ of $Q_0(\cdot)$. Then in a sense we apply the "sample analogue" principle, and obtain $Q_n(\cdot)$ as the sample analogue of $Q_0(\cdot)$, and $\hat \theta_n$ as the sample analogue of $\theta_0$. And then we examine what additional conditions may be needed in order to guarantee the desired convergence of the sample analogue to the population counterpart.

What is left is the assumption "$Q_n(\cdot)$ converges uniformly in probability to $Q_0(\cdot)$". To assume convergence per se appears naturally necessary -if $Q_n(\cdot)$ did not converge, or if it converged to some other function than $Q_0(\cdot)$, how could we relate $\hat \theta_n$ and $\theta_0$? But why do we need uniform convergence, and not just the weaker pointwise convergence?

Pointwise convergence (in probability) is defined as

$$\lim_{n\rightarrow \infty}P\left(\Big|Q_n(\theta)-Q_0(\theta)\Big|< \epsilon\right) =1,\;\;\epsilon>0,\;\; \forall\, \theta \in \Theta \tag{1}$$

while Uniform convergence as

$$\lim_{n\rightarrow \infty}P\left(\sup_{\theta \in \Theta}\Big|Q_n(\theta)-Q_0(\theta)\Big|< \epsilon\right) =1,\;\;\epsilon>0 \tag{2}$$

In pointwise convergence, we fix $\theta$ and then consider what happens as $n\rightarrow \infty$. If the limiting probability result holds for all $\theta$ (but each examined separately) then we have pointwise convergence.

In Uniform convergence, we form a sequence with the suprema of the function, and examine whether this sequence converges. And as $n$ changes, $\hat \theta_n$ in general changes, a fact that places what we are interested in studying outside the scope of pointwise convergence. Under pointwise convergence, we could not even write the probability statement using $\hat \theta_n$ -since the argument of $Q_n(\cdot)$ should remain fixed.

So we could say that the assumption of uniform convergence is not a matter of "how strong we need the convergence to be" - but, to begin with, a matter of assuming a form of convergence in the context of which we could indeed examine what we want to examine.

Up to now therefore, all assumptions made (and some regarding measurability with respect to the data that have been left unstated), appear mostly to be assumptions that permit us to consider whether we have convergence of $\hat \theta_n$ to $\theta_0$, and perhaps to have a "reasonable belief that we may obtain convergence"... They do not appear to be assumptions that guarantee (are sufficient for) the desired convergence result...

The impressive thing about it, is that these assumptions are also sufficient for consistency of the extremum estimator $\hat \theta_n$.