Question about Casella and Berger's proof of MLE invariance In Casella and Berger, p. 320, they have a proof of the invariance of the MLE. Let $g: \theta \mapsto \eta$ be a function. They define the induced likelihood as
$$
L^*(\eta \mid X) = \sup_{\{\theta: g(\theta) = \eta\}} L(\theta \mid X).
$$
This ensures that $g$ is one-to-one with respect to the likelihood functions because if $g$ is not one-to-one, there may be multiple values of $\theta$ that map to a given $g(\theta) = \eta$.
Then they note
$$
\sup_{\eta} L^*(\eta \mid X)
=
\sup_{\eta} \sup_{\{\theta: g(\theta) = \eta\}} L(\theta \mid X)
=
\sup_{\theta} L(\theta \mid X).
$$
The first equality just applies the definition of the MLE of $\eta$. However, the next inequality confuses me. They write,

The second inequality follows because the iterated maximization is equal to the unconditional maximization over $\theta$...

Can someone justify this claim or provide some intuition if the claim is supposed to obviously follow from definitions?
 A: The occurrences of suprema (instead of maxima, which might not exist) are troublesome.  Let us therefore isolate the basic underlying idea and rigorously establish it.
Definitions
Suppose $f:\Theta\to\mathbb{R}$ is any real-valued function on a set $\Theta.$  By definition, its supremum is the least upper bound of the values of $f:$
$$\sup_{\theta\in\Theta} f(\theta) = \operatorname{lub}\, \{f(\theta)\mid \theta\in\Theta\}.$$
As a shorthand, I will write $f^{*}_\Theta$ for this supremum.
The least upper bound of a set of real numbers $\mathcal A,$ written $\operatorname{lub}\,\mathcal A,$ is a number $x\in \mathbb{R}\cup \{\pm\infty\}$ (having the obvious ordering relation) with two defining properties (which, according to the axioms of Real numbers, make it unique):

*

*For all $a\in\mathcal A,$ $a \le x.$


*If $y$ is any number in $\mathbb{R}\cup \{\pm\infty\}$ satisfying (1), then $y \ge x.$
The Underlying Idea
Let $\Theta= \bigcup_{\mathcal A \in \mathbf{A}} \mathcal A$ be a union of sets.  For each such $\mathcal A$ let $f_{\mathcal A}$ be the restriction of $f$ to $\mathcal A.$  The claim is

$$\sup_{\mathcal A \in \mathbf{A}} f^{*}_{\mathcal A} = f^{*}_\Theta.$$

This is demonstrated in two steps.
First, when we assemble a bunch of suprema of $f$ over subsets of $\Theta,$ they cannot exceed the supremum of $f$ on $\Theta.$  Indeed, consider a set $\mathcal A\in \mathbf A.$ Because $\mathcal A$ is a subset of $\Theta,$ none of its elements exceed $f^{*}_\Theta.$ Consequently (by part (2) of the definition) $f^{*}_{\mathcal A} \le f^{*}_\Theta.$ A fortiori, $f^{*}_\Theta$ is an upper bound of all the $f^{*}_{\mathcal A},$ proving that
$$\sup_{\mathcal A \in \mathbf{A}} f^{*}_{\mathcal A} \le f^{*}_\Theta.\tag{*}$$
Second, let $y$ be an upper bound for all the $f^{*}_{\mathcal A}$ and let $\theta\in\Theta.$ Because $\Theta= \bigcup \mathcal A,$ there exists an $\mathcal A$ for which $\theta\in\mathcal A.$ Because $y \ge f^{*}_{\mathcal A},$ $y \ge \theta.$  Therefore (by part (2) of the definition), $y \ge f^{*}_\Theta.$  Because all upper bounds of the $f^{*}_{\mathcal A}$ exceed $f^{*}_\Theta,$
$$\sup_{\mathcal A \in \mathbf{A}} f^{*}_{\mathcal A} \ge f^{*}_\Theta.\tag{**}$$
The statements $(*)$ and $(**)$ prove the claim.
Application to Maximizing Likelihoods
The likelihood $\mathcal L$ is a function on a set $\Theta$ of distributions.  (I drop the reference to the data $X$ because $X$ will never change during this discussion.)  Given another function $g$ on this set, $\Theta$ can be expressed as the union of its level sets,
$$\Theta = \bigcup_{\eta\in\mathbb R} g^{-1}(\eta) = \bigcup_{\mathcal A \in \mathbf A} \mathcal A$$
where $\mathbf A$ is this collection of level sets.  In terms of the notation used in the question, our previously proven claim is the middle equality in
$$\sup_{\eta\in\mathbb R} \mathcal L^{*}(\eta) =\sup_{\eta\in\mathbb R} \mathcal L^{*}_{g^{-1}(\eta)} = \mathcal L^{*}_\Theta = \sup_{\theta\in\Theta}\mathcal{L}(\theta),$$
precisely as stated in the question.
Conclusions
This relationship between the "induced likelihood" and likelihood has nothing whatsoever to do with properties of likelihood, random variables, or anything else statistical: it is purely a statement about upper bounds of values attained by a function on a set.  The least upper bound can be defined with respect to the entire set $(\mathcal{L}^{*}_\Theta)$ or it can be found in stages by first taking the least upper bounds of subsets of the set $(\mathcal{L}^{*}_{g^{-1}(\eta)})$ and then finding the least upper bound of those upper bounds.
