# Induced Likelihood Function for Max Likelihood Estimators

A small doubt on the notation for the induced likelihood function and invariance properties of maximum likelihood estimators - It says in a textbook I am reading that:

Let $$X = {X1, X2,..., Xn}$$ be a random sample parameterized by $$θ$$. Suppose $$φ = g(θ)$$ where $$g : Θ → Φ$$. The induced likelihood function of $$φ$$ given an observed sample $$x$$ is:

$$L'$$X$$(φ; x) = sup$${θ:g(θ)=φ}$$L$$X$$(θ; x)$$

The book then proceeds to prove the invariance of MLEs by proving why $$L'$$X$$(\hat{φ}; x)$$ = $$L'$$X$$(g(\hat{θ}); x)$$.

So my questions are:

i) Is the apostrophe in $$L'$$X$$(φ; x)$$ supposed to denote the differential of $$L'$$X or just a variant of $$L$$X?

ii) In the first equation, shouldn't it be $$L'$$X$$(\hat{φ}; x)$$ instead of $$L'$$X$$(φ; x)$$? Why is $$sup$${θ:g(θ)=φ}$$L$$X$$(θ; x)$$ equal to the likelihood of $$x$$ for just any $$φ$$?

iii) What is the purpose of proving the second equation to support the first one? Haven't we already defined $$φ$$ as $$g(θ)$$?

i) Yes the apostrophe is here to emphasize that these are two differents functions (that may return differents values).

Take for example $$X \sim N(\theta, 1)$$ and $$g(\theta) = \theta-1$$ then $$L_X(1,x) = \frac{1}{\sqrt{2 \pi}} \exp \left ( -\frac{(x-1)^2}{2} \right )$$ while $$L'_X(1,x) = \frac{1}{\sqrt{2 \pi}} \exp \left ( -\frac{(x-2)^2}{2} \right )$$

thus $$L'_X(1,x) \neq L_X(1,x)$$.

ii) In the first equation the function $$L'_X$$ is defined w.r.t the parameter $$\phi$$ while $$\hat \phi$$ is the value that maximizes $$L'_X$$ (if such value exists).

For the question

Why is sup{θ:g(θ)=φ}LX(θ;x) equal to the likelihood of x for just any φ?

This is not a stated equality it is a definition, meaning that $$L'_X(\phi, x)$$ is defined that way. This in order to avoid the cases where two distincts values of $$\theta$$ lead to the same $$\phi$$: i.e if we have $$g(\theta_1) = g(\theta_2) = \phi$$ how do we define the likelihood regarding the "transformed" parameter $$\phi$$? A proposal is to take $$\sup_{\theta : g(\theta) = \phi} L_X(\theta,x)$$.

This is needed when for example the transform $$\theta \mapsto \phi$$ is not bijective. This is not the case when $$g$$ is invertible since there will be only one value $$\theta$$ for which $$g(\theta) = \phi$$. In that case $$L'_X(\phi, x) = L_X(g^{-1}(\phi),x)$$.

iii) Now we want to find the value $$\hat \phi$$ such that $$L'_X(\hat \phi,x)$$ reaches its maximum. That is $$\hat \phi := \text{arg}\max_\phi L'_X(\phi, x) = \text{arg}\max_\phi \sup_{\theta : g(\theta) = \phi} L_X(\theta, x)$$

If $$g$$ is invertible this reduces to find $$\hat \phi := \text{arg}\max_\phi L'_X(\phi, x) = \text{arg}\max_\phi L_X(g^{-1}(\phi), x)$$

In that case, since the MLE $$\hat \theta$$ is defined as $$\hat \theta = \text{arg}\max_\theta L_X(\theta, x)$$

we have $$g^{-1}(\hat \phi ) = \hat \theta$$ and thus $$\hat \phi = g(\hat \theta)$$.

The invariance of MLE states that this holds even when $$g$$ is not invertible anymore, i.e the value that maximizes $$L'_X(\phi,x)$$ is $$g(\hat \theta)$$ where $$\hat \theta$$ is the value that maximizes $$L_X(\theta,x)$$. This is a property of the induced likelihood that holds because of the way it has been defined.

Edit:

We have $$L'_X( \hat \phi , x) = \sup_\phi L'(\phi , x) = \sup_\phi \sup_{\theta : g(\theta) = \phi} L_X(\theta , x)$$ One "property" of the maximization process of a function is that

$$\sup_y \sup_{x \in R_y} f(x) = \sup_{x \in \cup_y R_y} f(x)$$

Applying this yield

$$\sup_\phi \sup_{\theta : g(\theta) = \phi} L_X(\theta , x) = \sup_\theta L_X(\theta, x) = L_X(\hat \theta, x)$$

Thus we have $$L'_X(\hat \phi, x) = L_X(\hat \theta, x) \qquad (1)$$

Finally we want to show that $$\hat \phi = g(\hat \theta)$$.

Since $$\hat \theta$$ is such that $$\sup_\theta L_X(\theta, x) = L(\hat \theta ,x )$$ , for any function $$f$$ we have,

$$L_X(\hat \theta , x) = \sup_{\theta : f(\theta) = f(\hat \theta)} L_X(\theta,x)$$

since $$\hat \theta \in \left \{ \theta : f(\theta ) = f(\hat \theta ) \right \}$$.

In particular if $$f=g$$ we have

\begin{align*} L_X(\hat \theta , x) &= \sup_{ \theta : g(\theta ) = g(\hat \theta) } L_X(\theta, x) \\ &= L'_X(g(\hat \theta) , x) \qquad (2) \end{align*}

And finally combining $$(1)$$ and $$(2)$$ shows that the MLE for $$L'_X$$, $$\hat \phi$$, is $$g(\hat \theta)$$.

• Hi @winperikle, thank you so much for the clarification. I know fully understand (i) and (ii), but am still a little unsure about (iii). The textbook I am reading goes on to actually prove the invariance when g is no longer invertible, but in a rather complex way. It says that Lx(θ^;x) = sup {θ∈Θ} Lx(θ;x) = sup {θ:g(θ)=g(θ^)} Lx(θ; x) = L'x(g(θ^); x). Mar 25, 2021 at 4:10
• I know this a bit unclear (but it's not allowing me to use Latex in the comments). Basically, it says that the likelihood of the estimator ϕ is equal to the likelihood of the estimator θ. This likelihood is equal to the supremum of the likelihood of just θ (w.r.t. to θ∈Θ). And this is, in turn, equal to the the same likelihood, but now w.r.t. to θ:g(θ)=g(θ^). And finally, this equals to the likelihood of the estimator g(θ^). Mar 25, 2021 at 4:10
• Do you have any idea of why changing the w.r.t. of the likelihood of just θ changed the equation to the likelihood of the estimator g(θ^)? Mar 25, 2021 at 4:10
• @Academic005 I have edited my answer, I hope that helps a bit. Mar 25, 2021 at 13:57