7
$\begingroup$

In Casella and Berger, p. 320, they have a proof of the invariance of the MLE. Let $g: \theta \mapsto \eta$ be a function. They define the induced likelihood as

$$ L^*(\eta \mid X) = \sup_{\{\theta: g(\theta) = \eta\}} L(\theta \mid X). $$

This ensures that $g$ is one-to-one with respect to the likelihood functions because if $g$ is not one-to-one, there may be multiple values of $\theta$ that map to a given $g(\theta) = \eta$.

Then they note

$$ \sup_{\eta} L^*(\eta \mid X) = \sup_{\eta} \sup_{\{\theta: g(\theta) = \eta\}} L(\theta \mid X) = \sup_{\theta} L(\theta \mid X). $$

The first equality just applies the definition of the MLE of $\eta$. However, the next inequality confuses me. They write,

The second inequality follows because the iterated maximization is equal to the unconditional maximization over $\theta$...

Can someone justify this claim or provide some intuition if the claim is supposed to obviously follow from definitions?

$\endgroup$
1
  • 5
    $\begingroup$ You're reading way too much into this. The statement is an arithmetic triviality. If you divide the domain of a function into pieces, find its largest value on each piece, and then take the largest of those largest values, you will have found the largest value of the function. $\endgroup$
    – whuber
    Dec 1, 2019 at 16:01

1 Answer 1

4
$\begingroup$

The occurrences of suprema (instead of maxima, which might not exist) are troublesome. Let us therefore isolate the basic underlying idea and rigorously establish it.

Definitions

Suppose $f:\Theta\to\mathbb{R}$ is any real-valued function on a set $\Theta.$ By definition, its supremum is the least upper bound of the values of $f:$

$$\sup_{\theta\in\Theta} f(\theta) = \operatorname{lub}\, \{f(\theta)\mid \theta\in\Theta\}.$$

As a shorthand, I will write $f^{*}_\Theta$ for this supremum.

The least upper bound of a set of real numbers $\mathcal A,$ written $\operatorname{lub}\,\mathcal A,$ is a number $x\in \mathbb{R}\cup \{\pm\infty\}$ (having the obvious ordering relation) with two defining properties (which, according to the axioms of Real numbers, make it unique):

  1. For all $a\in\mathcal A,$ $a \le x.$

  2. If $y$ is any number in $\mathbb{R}\cup \{\pm\infty\}$ satisfying (1), then $y \ge x.$

The Underlying Idea

Let $\Theta= \bigcup_{\mathcal A \in \mathbf{A}} \mathcal A$ be a union of sets. For each such $\mathcal A$ let $f_{\mathcal A}$ be the restriction of $f$ to $\mathcal A.$ The claim is

$$\sup_{\mathcal A \in \mathbf{A}} f^{*}_{\mathcal A} = f^{*}_\Theta.$$

This is demonstrated in two steps.

First, when we assemble a bunch of suprema of $f$ over subsets of $\Theta,$ they cannot exceed the supremum of $f$ on $\Theta.$ Indeed, consider a set $\mathcal A\in \mathbf A.$ Because $\mathcal A$ is a subset of $\Theta,$ none of its elements exceed $f^{*}_\Theta.$ Consequently (by part (2) of the definition) $f^{*}_{\mathcal A} \le f^{*}_\Theta.$ A fortiori, $f^{*}_\Theta$ is an upper bound of all the $f^{*}_{\mathcal A},$ proving that

$$\sup_{\mathcal A \in \mathbf{A}} f^{*}_{\mathcal A} \le f^{*}_\Theta.\tag{*}$$

Second, let $y$ be an upper bound for all the $f^{*}_{\mathcal A}$ and let $\theta\in\Theta.$ Because $\Theta= \bigcup \mathcal A,$ there exists an $\mathcal A$ for which $\theta\in\mathcal A.$ Because $y \ge f^{*}_{\mathcal A},$ $y \ge \theta.$ Therefore (by part (2) of the definition), $y \ge f^{*}_\Theta.$ Because all upper bounds of the $f^{*}_{\mathcal A}$ exceed $f^{*}_\Theta,$

$$\sup_{\mathcal A \in \mathbf{A}} f^{*}_{\mathcal A} \ge f^{*}_\Theta.\tag{**}$$

The statements $(*)$ and $(**)$ prove the claim.

Application to Maximizing Likelihoods

The likelihood $\mathcal L$ is a function on a set $\Theta$ of distributions. (I drop the reference to the data $X$ because $X$ will never change during this discussion.) Given another function $g$ on this set, $\Theta$ can be expressed as the union of its level sets,

$$\Theta = \bigcup_{\eta\in\mathbb R} g^{-1}(\eta) = \bigcup_{\mathcal A \in \mathbf A} \mathcal A$$

where $\mathbf A$ is this collection of level sets. In terms of the notation used in the question, our previously proven claim is the middle equality in

$$\sup_{\eta\in\mathbb R} \mathcal L^{*}(\eta) =\sup_{\eta\in\mathbb R} \mathcal L^{*}_{g^{-1}(\eta)} = \mathcal L^{*}_\Theta = \sup_{\theta\in\Theta}\mathcal{L}(\theta),$$

precisely as stated in the question.

Conclusions

This relationship between the "induced likelihood" and likelihood has nothing whatsoever to do with properties of likelihood, random variables, or anything else statistical: it is purely a statement about upper bounds of values attained by a function on a set. The least upper bound can be defined with respect to the entire set $(\mathcal{L}^{*}_\Theta)$ or it can be found in stages by first taking the least upper bounds of subsets of the set $(\mathcal{L}^{*}_{g^{-1}(\eta)})$ and then finding the least upper bound of those upper bounds.

$\endgroup$
3
  • $\begingroup$ Is $\bigcup_{\eta\in\mathbb{R}}g^{-1}(\eta)$ a shorthand notation for $\bigcup_{\eta\in\mathbb{R}}\left\{ \theta\in\Theta:g(\theta)=\eta\right\}$? $\endgroup$
    – statmerkur
    Feb 7, 2021 at 19:13
  • $\begingroup$ @stat Yes, the definition of the level set is that $g^{-1}(\eta)=\{\theta\in\Theta\mid g(\theta)=\eta\}.$ $\endgroup$
    – whuber
    Feb 7, 2021 at 19:15
  • 1
    $\begingroup$ Thank you for the clarification. I wasn't aware of this notation / am used to a different notation for level sets but this one also makes sense to me since a level set is the preimage of a singleton under a function. I would probably denote this by $g^{-1}\left(\left\{ \eta\right\} \right)$ but now that I've thought about it it's clear what $g^{-1}\left(\eta\right)$ refers to. Btw +1 for this nice exposition. $\endgroup$
    – statmerkur
    Feb 7, 2021 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.