8
$\begingroup$

If you have a measurable parameter space $(\Theta, \mathcal{F})$ and a parametric family of probability measures $(P_\theta)_{\theta \in \Theta}$ on a measurable space $(\mathcal{X}, \mathcal{B})$ that have densities $(f_\theta)_{\theta \in \Theta}$ with respect to some dominating measure $\mu$, then you might define the maximum likelihood estimator $\hat{\theta} : \mathcal{X} \to \Theta$ as $$ \tag{1}\label{1} \hat{\theta}(x) = \operatorname*{arg\,max}_{\theta \in \Theta} f_\theta(x) $$ for all $x \in \mathcal{X}$.

This is an abstract version of how MLEs are introduced in some statistics texts and classes, but there are a few issues. For one thing, a global maximum might not be attained, or there might be multiple global maxima, so \eqref{1} isn't necessarily well-defined. Moreover, if it happens to be well-defined, then is it measurable? Certainly all examples of maximum likelihood estimators one sees in applied statistics courses (at least that I've seen) are measurable.

Of course you can get around these issues by defining MLEs in a different way: for example, you might call a function $\hat{\theta} : \mathcal{X} \to \Theta$ a maximum likelihood estimator if it is a priori measurable and there exists a set $N \in \mathcal{B}$ such that $\mu(N) = 0$ and $$ \tag{2}\label{2} f_{\hat{\theta}(x)}(x) = \sup_{\theta \in \Theta} f_\theta(x) $$ for all $x \in \mathcal{X} \setminus N$.

However, oftentimes \eqref{2} isn't how maximum likelihood estimation is introduced, and I'm more interested in the question of measurability of functions $\hat{\theta}$ satisfying \eqref{1} or \eqref{2}.


Questions. Are there any explicit examples in which some reasonable interpretation of \eqref{1} is well-defined (at least off a $\mu$-null set) but not measurable? Alternatively, can a function $\hat{\theta}$ satisfying \eqref{2} (again, at least off a $\mu$-null set) exist when no such measurable function exists? What about if $(\Theta, \mathcal{F})$ and $(\mathcal{X}, \mathcal{B})$ are standard Borel spaces? If there isn't a constructive example, then is there a proof of the existence of an example?


There are various results that give criteria under which there cannot be examples of the kind I'm looking for, thereby restricting the class of spaces and densities that might possess such examples.

Example 1. Lemma 2 in Jennrich (1969) implies that if

  • $\Theta$ is a compact subset of a Euclidean space and $\mathcal{F}$ is its Borel $\sigma$-algebra, and
  • the function $\theta \mapsto f_\theta(x)$ from $\Theta$ into $\mathbb{R}$ is continuous for every $x \in \mathcal{X}$,

then there exists a measurable function $\hat{\theta} : \mathcal{X} \to \Theta$ such that \eqref{2} holds for all $x \in \mathcal{X}$.

Example 2. Theorem 3.1 in Wagner (1980) (the Kuratowski and Ryll-Nardzewski selection theorem) implies that if

  • $(\Theta, \mathcal{F})$ is a Polish space with its Borel $\sigma$-algebra,
  • $(\mathcal{X}, \mathcal{B})$ is any measurable space,
  • for each $x \in \mathcal{X}$, the set $$ F(x) = \big\{\theta \in \Theta : \text{$f_\theta(x) \geq f_\psi(x)$ for all $\psi\in\Theta$}\big\} $$ is nonempty and closed, and
  • for every open $U \subseteq \Theta$, the set $$ \big\{x \in \mathcal{X} : F(x) \cap U \neq \emptyset\big\} $$ belongs to $\mathcal{B}$,

then there exists a measurable function $\hat{\theta} : \mathcal{X} \to \Theta$ such that $\hat{\theta}(x) \in F(x)$ for all $x \in \mathcal{X}$ (i.e., \eqref{2} holds for all $x \in \mathcal{X}$).

Thus, under the additional assumptions of Example 1 or Example 2 there always exists a measurable maximum likelihood estimator.

Example 3. Theorem 7.50(b) in Bertsekas and Shreve (1996) implies that if

  • $(\Theta, \mathcal{F})$ and $(\mathcal{X}, \mathcal{B})$ are standard Borel spaces, and
  • for every $c \in \mathbb{R}$, the set $$ \big\{ (x, \theta) \in \mathcal{X} \times \Theta : f_\theta(x) > c \big\} $$ is an analytic subset of $\mathcal{X} \times \Theta$,

then the set $$ I = \left\{ x \in \mathcal{X} : \text{$f_\theta(x) = \sup_{\psi\in\Theta} f_\psi(x)$ for some $\theta \in \Theta$}\right\} $$ is universally measurable. Moreover, for each $\varepsilon > 0$ there exists a universally measurable function $\hat{\theta} : \mathcal{X} \to \Theta$ such that \eqref{2} holds for all $x \in I$, and for all $x \in \mathcal{X} \setminus I$, $$ f_{\hat{\theta}(x)}(x) \geq \begin{cases} \sup_{\psi \in \Theta} f_\psi(x) - \varepsilon, & \text{if $\sup_{\psi \in \Theta} f_\psi(x) < \infty$,} \\ \varepsilon^{-1}, & \text{if $\sup_{\psi \in \Theta} f_\psi(x) = \infty$.} \end{cases} $$

Example 3 doesn't necessarily guarantee the existence of a measurable maximum likelihood estimator (for instance, since there are universally measurable sets which are not Borel), but it is arguably in the same spirit as Example 2.

$\endgroup$
  • 1
    $\begingroup$ Since a measurable function is defined between two measurable spaces, the parameter space $\Theta$ would need to be endowed with a natural measure for the question to be well-defined. $\endgroup$ – Xi'an Oct 11 '19 at 6:52
  • $\begingroup$ @Xi'an there is no measure involved in the definition of a measurable map (or a measurable space for that matter). In the measure-theoretic development of statistics, estimators need to be measurable functions. Otherwise we couldn't talk about distributions of estimators. $\endgroup$ – Artem Mavrin Oct 11 '19 at 7:01
  • $\begingroup$ Any text that proposes $\Theta$ is a measure space is going to present a Bayesian theory rather than MLEs. (Classical texts do not make any such assumption because they do not always need to model the parameter as a random variable.) In this sense your characterization of the situation appears not to reflect anything one would find in any account of MLE. $\endgroup$ – whuber Oct 11 '19 at 15:04
  • $\begingroup$ @whuber I’m not assuming $\Theta$ is a measure space, merely that it’s endowed with a $\sigma$-algebra. In most situations in practice $\Theta$ is a Borel subset of a Euclidean space, so it comes with a natural $\sigma$-algebra that doesn’t need to be mentioned. If the parameter space isn’t endowed with a $\sigma$-algebra then we can’t talk about parameter-valued estimators $\endgroup$ – Artem Mavrin Oct 11 '19 at 15:09
  • 1
    $\begingroup$ My sense is it's possible, but imposing "regularity conditions" on the parameterization might help ensure measurability. That means the parameterization has to be continuous relative to some reasonable topology on the space of probability distributions. $\endgroup$ – whuber Oct 11 '19 at 17:41
3
$\begingroup$

Here is a contrived example.

Let $(\mathcal{X}, \mathcal{B})$ be the interval $[0, 2]$ with its Borel $\sigma$-algebra, let $(\Theta, \mathcal{F})$ be the interval $[1, 2]$ with its Borel $\sigma$-algebra, and let $P_\theta$ be the uniform distribution on $[0, \theta]$ for each $\theta \in \Theta$. The family $(P_\theta)_{\theta \in \Theta}$ is dominated by the restriction of Lebesgue measure to $\mathcal{X}$ (call it $\mu$ as in the question), and a sensible choice of density for $P_\theta$ with respect to $\mu$ is $$ f_\theta = \theta^{-1} \mathbf{1}_{[0, \theta]}. $$ The maximum likelihood estimator $\hat{\theta} : \mathcal{X} \to \Theta$ is then given by $$ \hat{\theta}(x) = \max\{1, x\}, $$ which is certainly measurable.

However, suppose I'm not sensible, and that I have a favorite Vitali set $V \subseteq [0, 1]$, and I define a family $(g_\theta)_{\theta \in \Theta}$ of functions $\mathcal{X} \to \mathbb{R}$ as follows:

  • if $\theta - 1\in V$, then $g_\theta = 3 \mathbf{1}_{\{\theta - 1\}} + \theta^{-1} \mathbf{1}_{[0, \theta] \setminus \{\theta - 1\}}$;
  • if $\theta - 1\notin V$, then $g_\theta = f_\theta$.

In other words, $$ g_\theta(x) = \begin{cases} 3, & \text{if $x \in V$ and $x = \theta - 1$,} \\ f_\theta(x), & \text{otherwise} \end{cases} $$ for all $x \in \mathcal{X}$ and $\theta \in \Theta$. Since each $g_\theta$ is a modification of $f_\theta$ on at most a singleton, the family $(g_\theta)_{\theta \in \Theta}$ is another family of densities for $(P_\theta)_{\theta \in \Theta}$ with respect to $\mu$.

Now one can check that there is a unique function $\hat{\vartheta} : \mathcal{X} \to \Theta$ such that $$ g_{\hat{\vartheta}(x)}(x) = \sup_{\theta \in \Theta} g_\theta(x) $$ for each $x \in \mathcal{X}$; this function is given by $$ \hat{\vartheta}(x) = \begin{cases} \max\{1, x\}, & \text{if $x \notin V$,} \\ x + 1, & \text{if $x \in V$,} \end{cases} $$ and it is not measurable (for example, since $\{x \in \mathcal{X} : \hat{\vartheta}(x) = x + 1\} = \{0\} \cup V$).

Thus, in the setup with the family of densities $(g_\theta)_{\theta \in \Theta}$, there exists a function $\hat{\vartheta} : \mathcal{X} \to \Theta$ such that $$ g_{\hat{\vartheta}(x)}(x) = \sup_{\theta \in \Theta} g_\theta(x) $$ for all $x \in \mathcal{X}$, but no such measurable function exists.

This example merely shows that the the question of whether a measurable maximum likelihood estimator exists is sensitive to the choice of version of the Radon-Nikodym derivative $dP_\theta / d\mu$. When we used the much more reasonable family of densities $(f_\theta)_{\theta \in \Theta}$ we had no trouble obtaining a measurable maximum likelihood estimator, but when we artificially perturbed each of these densities on a null set to obtain $(g_\theta)_{\theta \in \Theta}$, we ran into problems.

This example is inspired by Example 2.2 in Pfanzagl (1969), wherein a similar but much more subtle modification of densities yields a scenario in which no maximum likelihood estimator exists at all, measurable or not.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.