# Meaning of Invariance of Maximum Likelihood Estimator

In Casella-Berger, the invariance of MLE is defined as:

Assuming that $$\hat{\theta}$$ is MLE of $$\theta$$, then for any function $$\tau$$, $$\tau(\hat{\theta})$$ is MLE of $$\tau(\theta)$$.

In the case of a one-to-one transformation, everything is clear.

But what is the point of having $$\tau$$ not one-to-one? In C&B, there is even a proof of this fact. However, the authors defined a NEW LIKELIHOOD, called "induced likelihood function" (profile likelihood), which is defined as

$$\mathcal{L}^*(\eta|x) = \sup_{\{\theta: \tau(\theta) = \eta\}}\mathcal{L}(\theta|x)$$

Sure, we can prove that $$\tau(\hat{\theta})$$ maximizes the profile likelihood above. But this is NOT THE SAME LIKELIHOOD as in the original problem!

Example: imagine $$f(x|\theta) = normal(\theta, 1)$$, and a transformation $$\tau(\theta) = \theta^2$$. Ok, $$\tau(\hat{\theta})$$ maximises the profile likelihood. Now what?

I guess I would summarize my question:

1. What is the point of the whole theorem in the case of non-bijective functions $$\tau$$.

2. Assuming $$\eta$$ maximizes the profile likelihood, what does it tell me about the original likelihood? E.g. in the example above, I cannot convert $$\eta = \theta^2$$ back into $$\theta$$.

Casella and Berger (2002) explain this by saying that, "In many cases, this simple version of the invariance of MLEs is not useful because many of the functions we are interested in are not one-to-one" (p. 320). This is the essence of their motivation for extending the concept of the likelihood function. If you have an existing sampling density framed in terms of a minimal sufficient parameter, and you want an estimate of a non-injective function of that parameter (as in your example) then the latter quantity is not a valid parametric to describe the sampling density.

One of the examples they give (on p. 321) which is quite instructive, is the case where you have binomial data $$X \sim \text{Bin}(n,p)$$ and you want to make an inference about the standard deviation $$\mathbb{S}(X) = \sqrt{np(1-p)}$$. This is a valuable quantity of interest, but it is a non-injective function of the parameter $$p$$. (In particular, we get the same standard deviation for $$p$$ and $$p' = 1-p$$.) This is the kind of situation where it is useful to find the MLE for a non-injective function of a parameter.

This is not so different from standard MLE analysis: The situation in your example is not actually any different from the case where we have a distribution that depends on two parameters, and we only want an MLE for one of those parameters. In fact, you can re-frame it directly in those terms. Taking your example, suppose we reframe the sampling density in terms of the two parameters $$\phi \geqslant 0$$ and $$\lambda \in \{ -1, 1\}$$ and we use the sampling density:

$$f(x|\phi, \lambda) = \text{N}(x|\phi \lambda, 1) = \frac{1}{\sqrt{2 \pi}} \exp \Big( -\frac{(x-\phi \lambda)^2}{2} \Big),$$

and the corresponding likelihood function:

$$L_x(\phi, \lambda) \propto \exp \Big( -\frac{(x-\phi \lambda)^2}{2} \Big).$$

This new representation is just as valid as the original representation in terms of $$\theta = \phi \lambda$$, and all that we have done is to split this parameter into two parameters that capture aspects of its behaviour. In the new form, the parameter $$\tau = \theta^2 = \phi^2$$ is a one-to-one function of our parameter $$\phi$$, so we can unambiguously re-frame our likelihood function as:

$$L_x(\tau, \lambda) \propto \exp \Big( -\frac{(x-\sqrt{\tau} \lambda)^2}{2} \Big).$$

Conseqeuently, the MLE for the parameter $$\tau$$ is any value that satisfies $$L_x(\hat{\phi}_\text{MLE}, \hat{\lambda}_\text{MLE}) \geqslant L_x(\phi, \lambda)$$ for all valid $$\phi$$ and $$\lambda$$. It can easily be shown that the maximised likelihood corresponds to:

\begin{align} L_x(\hat{\phi}_\text{MLE}, \hat{\lambda}_\text{MLE}) = \sup_{\lambda \in \{-1,1 \} } L_x(\hat{\phi}_\text{MLE}, \lambda) = L_x^*(\hat{\phi}_\text{MLE}), \\[6pt] \end{align}

using the "induced likelihood" defined in Casella and Berger. So, you can see that you are already using this technique when you look at the MLE of a parameter in a context where you have a model that depends on multiple parameters. The situation is not fundamentally different if you have data $$X \sim \text{N}(\theta, \sigma^2)$$ and you want the MLE for $$\theta$$, while treating $$\sigma$$ as a nuisance parameter. In each case the MLE of the parameter of interest maximises the "induced likelihood".

• In your second paragraph (Binomial variance). Assuming I estimate the std $S(X)$ using MLE of $p$, and use the invariance theorem. Then what likelihood does S(X) satisfy exactly? What is the concrete likelihood function that $\sqrt{np(1-p)}$ satisfies as $argmax$? Commented Sep 3, 2021 at 9:00
• to clarify my above comment - the induced likelihood function is just an abstract construct, that has no obvious meaning. For example, when you maximize Binomial(x, n, p) for $p$, you are "maximizing the probability" of the observed sample $x$. When you get an estimate $\hat{\tau} = \sqrt{np(1-p)}$ for the "induced likelihood", there should exist a probability density, which is parameterized by $\hat{\tau}$, that corresponds to the induced likelihood, and related somehow to the binomial density, or there is no point talking about MLE for $\hat{\tau}$. Commented Sep 3, 2021 at 9:53
• Everything in mathematics is abstract concepts, and the induced likelihood does not seem especially difficult to interpret (maximum of the original likelihood taken over a parameter range). Your assertion that there is no point in getting/taking about an MLE unless underlying probabilistic conditions desired by you exist is false; the MLE is an estimator, and it may be useful to estimate a thing we want to know about. Try creating a bijective mapping from $(n,p)$ to $(\sqrt{np(1-p)}, \lambda)$ for some parameter $\lambda$, and you might get a density of the kind you are looking for.
– Ben
Commented Sep 3, 2021 at 20:30