1
$\begingroup$

A small doubt on the notation for the induced likelihood function and invariance properties of maximum likelihood estimators - It says in a textbook I am reading that:

Let $X = {X1, X2,..., Xn}$ be a random sample parameterized by $θ$. Suppose $φ = g(θ)$ where $g : Θ → Φ$. The induced likelihood function of $φ$ given an observed sample $x$ is:

$L'$X$(φ; x) = sup ${θ:g(θ)=φ}$ L$X$(θ; x)$

The book then proceeds to prove the invariance of MLEs by proving why $L'$X$(\hat{φ}; x)$ = $L'$X$(g(\hat{θ}); x)$.

So my questions are:

i) Is the apostrophe in $L'$X$(φ; x)$ supposed to denote the differential of $L'$X or just a variant of $L$X?

ii) In the first equation, shouldn't it be $L'$X$(\hat{φ}; x)$ instead of $L'$X$(φ; x)$? Why is $sup ${θ:g(θ)=φ}$ L$X$(θ; x)$ equal to the likelihood of $x$ for just any $φ$?

iii) What is the purpose of proving the second equation to support the first one? Haven't we already defined $φ$ as $g(θ)$?

Thanks in advance for your help!

$\endgroup$

1 Answer 1

3
$\begingroup$

i) Yes the apostrophe is here to emphasize that these are two differents functions (that may return differents values).

Take for example $X \sim N(\theta, 1)$ and $g(\theta) = \theta-1$ then $$ L_X(1,x) = \frac{1}{\sqrt{2 \pi}} \exp \left ( -\frac{(x-1)^2}{2} \right ) $$ while $$ L'_X(1,x) = \frac{1}{\sqrt{2 \pi}} \exp \left ( -\frac{(x-2)^2}{2} \right ) $$

thus $L'_X(1,x) \neq L_X(1,x)$.

ii) In the first equation the function $L'_X$ is defined w.r.t the parameter $\phi$ while $\hat \phi$ is the value that maximizes $L'_X$ (if such value exists).

For the question

Why is sup{θ:g(θ)=φ}LX(θ;x) equal to the likelihood of x for just any φ?

This is not a stated equality it is a definition, meaning that $L'_X(\phi, x)$ is defined that way. This in order to avoid the cases where two distincts values of $\theta$ lead to the same $\phi$: i.e if we have $g(\theta_1) = g(\theta_2) = \phi$ how do we define the likelihood regarding the "transformed" parameter $\phi$? A proposal is to take $\sup_{\theta : g(\theta) = \phi} L_X(\theta,x)$.

This is needed when for example the transform $\theta \mapsto \phi$ is not bijective. This is not the case when $g$ is invertible since there will be only one value $\theta$ for which $g(\theta) = \phi$. In that case $L'_X(\phi, x) = L_X(g^{-1}(\phi),x)$.

iii) Now we want to find the value $\hat \phi$ such that $L'_X(\hat \phi,x)$ reaches its maximum. That is $$ \hat \phi := \text{arg}\max_\phi L'_X(\phi, x) = \text{arg}\max_\phi \sup_{\theta : g(\theta) = \phi} L_X(\theta, x) $$

If $g$ is invertible this reduces to find $$ \hat \phi := \text{arg}\max_\phi L'_X(\phi, x) = \text{arg}\max_\phi L_X(g^{-1}(\phi), x) $$

In that case, since the MLE $\hat \theta$ is defined as $$ \hat \theta = \text{arg}\max_\theta L_X(\theta, x) $$

we have $g^{-1}(\hat \phi ) = \hat \theta$ and thus $\hat \phi = g(\hat \theta)$.

The invariance of MLE states that this holds even when $g$ is not invertible anymore, i.e the value that maximizes $L'_X(\phi,x)$ is $g(\hat \theta)$ where $\hat \theta$ is the value that maximizes $L_X(\theta,x)$. This is a property of the induced likelihood that holds because of the way it has been defined.


Edit:

We have $$ L'_X( \hat \phi , x) = \sup_\phi L'(\phi , x) = \sup_\phi \sup_{\theta : g(\theta) = \phi} L_X(\theta , x) $$ One "property" of the maximization process of a function is that

$$ \sup_y \sup_{x \in R_y} f(x) = \sup_{x \in \cup_y R_y} f(x) $$

Applying this yield

$$ \sup_\phi \sup_{\theta : g(\theta) = \phi} L_X(\theta , x) = \sup_\theta L_X(\theta, x) = L_X(\hat \theta, x) $$

Thus we have $$ L'_X(\hat \phi, x) = L_X(\hat \theta, x) \qquad (1) $$

Finally we want to show that $\hat \phi = g(\hat \theta)$.

Since $\hat \theta$ is such that $\sup_\theta L_X(\theta, x) = L(\hat \theta ,x )$ , for any function $f$ we have,

$$ L_X(\hat \theta , x) = \sup_{\theta : f(\theta) = f(\hat \theta)} L_X(\theta,x) $$

since $\hat \theta \in \left \{ \theta : f(\theta ) = f(\hat \theta ) \right \}$.

In particular if $f=g$ we have

\begin{align*} L_X(\hat \theta , x) &= \sup_{ \theta : g(\theta ) = g(\hat \theta) } L_X(\theta, x) \\ &= L'_X(g(\hat \theta) , x) \qquad (2) \end{align*}

And finally combining $(1)$ and $(2)$ shows that the MLE for $L'_X$, $\hat \phi$, is $g(\hat \theta)$.

$\endgroup$
4
  • $\begingroup$ Hi @winperikle, thank you so much for the clarification. I know fully understand (i) and (ii), but am still a little unsure about (iii). The textbook I am reading goes on to actually prove the invariance when g is no longer invertible, but in a rather complex way. It says that Lx(θ^;x) = sup {θ∈Θ} Lx(θ;x) = sup {θ:g(θ)=g(θ^)} Lx(θ; x) = L'x(g(θ^); x). $\endgroup$ Mar 25, 2021 at 4:10
  • $\begingroup$ I know this a bit unclear (but it's not allowing me to use Latex in the comments). Basically, it says that the likelihood of the estimator ϕ is equal to the likelihood of the estimator θ. This likelihood is equal to the supremum of the likelihood of just θ (w.r.t. to θ∈Θ). And this is, in turn, equal to the the same likelihood, but now w.r.t. to θ:g(θ)=g(θ^). And finally, this equals to the likelihood of the estimator g(θ^). $\endgroup$ Mar 25, 2021 at 4:10
  • $\begingroup$ Do you have any idea of why changing the w.r.t. of the likelihood of just θ changed the equation to the likelihood of the estimator g(θ^)? $\endgroup$ Mar 25, 2021 at 4:10
  • $\begingroup$ @Academic005 I have edited my answer, I hope that helps a bit. $\endgroup$
    – periwinkle
    Mar 25, 2021 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.