# Question about Casella and Berger's proof of MLE invariance

In Casella and Berger, p. 320, they have a proof of the invariance of the MLE. Let $$g: \theta \mapsto \eta$$ be a function. They define the induced likelihood as

$$L^*(\eta \mid X) = \sup_{\{\theta: g(\theta) = \eta\}} L(\theta \mid X).$$

This ensures that $$g$$ is one-to-one with respect to the likelihood functions because if $$g$$ is not one-to-one, there may be multiple values of $$\theta$$ that map to a given $$g(\theta) = \eta$$.

Then they note

$$\sup_{\eta} L^*(\eta \mid X) = \sup_{\eta} \sup_{\{\theta: g(\theta) = \eta\}} L(\theta \mid X) = \sup_{\theta} L(\theta \mid X).$$

The first equality just applies the definition of the MLE of $$\eta$$. However, the next inequality confuses me. They write,

The second inequality follows because the iterated maximization is equal to the unconditional maximization over $$\theta$$...

Can someone justify this claim or provide some intuition if the claim is supposed to obviously follow from definitions?

• You're reading way too much into this. The statement is an arithmetic triviality. If you divide the domain of a function into pieces, find its largest value on each piece, and then take the largest of those largest values, you will have found the largest value of the function.
– whuber
Dec 1, 2019 at 16:01

The occurrences of suprema (instead of maxima, which might not exist) are troublesome. Let us therefore isolate the basic underlying idea and rigorously establish it.

### Definitions

Suppose $$f:\Theta\to\mathbb{R}$$ is any real-valued function on a set $$\Theta.$$ By definition, its supremum is the least upper bound of the values of $$f:$$

$$\sup_{\theta\in\Theta} f(\theta) = \operatorname{lub}\, \{f(\theta)\mid \theta\in\Theta\}.$$

As a shorthand, I will write $$f^{*}_\Theta$$ for this supremum.

The least upper bound of a set of real numbers $$\mathcal A,$$ written $$\operatorname{lub}\,\mathcal A,$$ is a number $$x\in \mathbb{R}\cup \{\pm\infty\}$$ (having the obvious ordering relation) with two defining properties (which, according to the axioms of Real numbers, make it unique):

1. For all $$a\in\mathcal A,$$ $$a \le x.$$

2. If $$y$$ is any number in $$\mathbb{R}\cup \{\pm\infty\}$$ satisfying (1), then $$y \ge x.$$

### The Underlying Idea

Let $$\Theta= \bigcup_{\mathcal A \in \mathbf{A}} \mathcal A$$ be a union of sets. For each such $$\mathcal A$$ let $$f_{\mathcal A}$$ be the restriction of $$f$$ to $$\mathcal A.$$ The claim is

$$\sup_{\mathcal A \in \mathbf{A}} f^{*}_{\mathcal A} = f^{*}_\Theta.$$

This is demonstrated in two steps.

First, when we assemble a bunch of suprema of $$f$$ over subsets of $$\Theta,$$ they cannot exceed the supremum of $$f$$ on $$\Theta.$$ Indeed, consider a set $$\mathcal A\in \mathbf A.$$ Because $$\mathcal A$$ is a subset of $$\Theta,$$ none of its elements exceed $$f^{*}_\Theta.$$ Consequently (by part (2) of the definition) $$f^{*}_{\mathcal A} \le f^{*}_\Theta.$$ A fortiori, $$f^{*}_\Theta$$ is an upper bound of all the $$f^{*}_{\mathcal A},$$ proving that

$$\sup_{\mathcal A \in \mathbf{A}} f^{*}_{\mathcal A} \le f^{*}_\Theta.\tag{*}$$

Second, let $$y$$ be an upper bound for all the $$f^{*}_{\mathcal A}$$ and let $$\theta\in\Theta.$$ Because $$\Theta= \bigcup \mathcal A,$$ there exists an $$\mathcal A$$ for which $$\theta\in\mathcal A.$$ Because $$y \ge f^{*}_{\mathcal A},$$ $$y \ge \theta.$$ Therefore (by part (2) of the definition), $$y \ge f^{*}_\Theta.$$ Because all upper bounds of the $$f^{*}_{\mathcal A}$$ exceed $$f^{*}_\Theta,$$

$$\sup_{\mathcal A \in \mathbf{A}} f^{*}_{\mathcal A} \ge f^{*}_\Theta.\tag{**}$$

The statements $$(*)$$ and $$(**)$$ prove the claim.

### Application to Maximizing Likelihoods

The likelihood $$\mathcal L$$ is a function on a set $$\Theta$$ of distributions. (I drop the reference to the data $$X$$ because $$X$$ will never change during this discussion.) Given another function $$g$$ on this set, $$\Theta$$ can be expressed as the union of its level sets,

$$\Theta = \bigcup_{\eta\in\mathbb R} g^{-1}(\eta) = \bigcup_{\mathcal A \in \mathbf A} \mathcal A$$

where $$\mathbf A$$ is this collection of level sets. In terms of the notation used in the question, our previously proven claim is the middle equality in

$$\sup_{\eta\in\mathbb R} \mathcal L^{*}(\eta) =\sup_{\eta\in\mathbb R} \mathcal L^{*}_{g^{-1}(\eta)} = \mathcal L^{*}_\Theta = \sup_{\theta\in\Theta}\mathcal{L}(\theta),$$

precisely as stated in the question.

### Conclusions

This relationship between the "induced likelihood" and likelihood has nothing whatsoever to do with properties of likelihood, random variables, or anything else statistical: it is purely a statement about upper bounds of values attained by a function on a set. The least upper bound can be defined with respect to the entire set $$(\mathcal{L}^{*}_\Theta)$$ or it can be found in stages by first taking the least upper bounds of subsets of the set $$(\mathcal{L}^{*}_{g^{-1}(\eta)})$$ and then finding the least upper bound of those upper bounds.

• Is $\bigcup_{\eta\in\mathbb{R}}g^{-1}(\eta)$ a shorthand notation for $\bigcup_{\eta\in\mathbb{R}}\left\{ \theta\in\Theta:g(\theta)=\eta\right\}$? Feb 7, 2021 at 19:13
• @stat Yes, the definition of the level set is that $g^{-1}(\eta)=\{\theta\in\Theta\mid g(\theta)=\eta\}.$
– whuber
Feb 7, 2021 at 19:15
• Thank you for the clarification. I wasn't aware of this notation / am used to a different notation for level sets but this one also makes sense to me since a level set is the preimage of a singleton under a function. I would probably denote this by $g^{-1}\left(\left\{ \eta\right\} \right)$ but now that I've thought about it it's clear what $g^{-1}\left(\eta\right)$ refers to. Btw +1 for this nice exposition. Feb 7, 2021 at 19:49