1
$\begingroup$

I am reading the proof of the invariance property of MLE from Casella and Berger.

In this proof we parametrize : $\eta = \tau(\theta)$

There we define the induced likelihood function:

$ L_{1}^*(\eta|x) = sup_{\theta|\tau(\theta) = \eta} L(\theta|x) \tag{1}$

I have subscripted L*($\eta$|x) by 1 to differentiate between the induced likelihood of $\eta $ and the Likelihood of $\eta$ which are both denoted by $L^*(\eta|x)$

I am not sure why this is being done. (In what follows,L* is the likelihood of $\eta$ ). If $\theta_1$ and $\theta_2$ are such that $\tau(\theta_1) = \tau(\theta_2)$ then $L(\theta_1|x)$ = $L^*(\eta = \tau(\theta_1)|x)$= $L^*(\eta = \tau(\theta_2)|x)$ = $L(\theta_2|x$) since $\tau(\theta_1)$ =$\tau(\theta_2)$

Hence there is no need of the supremum in (1).

Where do I misunderstand?

Improve this question
$\endgroup$
6
  • $\begingroup$ What assumptions are you making about $\tau$? If they do not include injectivity, then consider a constant function. According to your argument, that would imply the likelihood $L$ is constant (since $\tau(\theta_1)=\tau(\theta_2)$ for all $\theta_1$ and $\theta_2$). $\endgroup$
    – whuber ♦
    Commented Sep 8, 2017 at 12:44
  • $\begingroup$ Suppose $\eta = \tau(\theta)$ = some constant say k. Then $Prob(x|\eta)$ would be a fixed number. All values of $\eta$ would be equally likely. Also, the Prob(x|$\theta$) = Prob(x|$\tau(\theta)$)= constant).ie. all values of $\theta $ are equally likely. There is no contradiction. $\endgroup$
    – user2338823
    Commented Sep 8, 2017 at 13:06
  • $\begingroup$ The contradiction is in your post itself, where you conclude that "$L(\theta_1|x)=\cdots=L(\theta_2|x)$", which will not be the case. $\endgroup$
    – whuber ♦
    Commented Sep 8, 2017 at 13:08
  • $\begingroup$ My conclusion is that,L($\theta_1|x$) = L($\theta_2|x$) IF they map to the same $\eta$.( I had written that $\tau(\theta_1)$ = $\tau(\theta_2)$. In the constant function example, $\eta$ is fixed and all $\theta$'s are equally likely, there is no contradiction. $\endgroup$
    – user2338823
    Commented Sep 8, 2017 at 13:30
  • 1
    $\begingroup$ In the constant function example, all $\theta$'s map to the same value. That's what it means to be constant! There's no contradiction only if the likelihood itself is a constant function. $\endgroup$
    – whuber ♦
    Commented Sep 8, 2017 at 15:02

2 Answers 2

Reset to default
2
$\begingroup$

Perhaps the issues here are best understood in the context of an example. Suppose that we are interested in estimating the mean of a normal model with variance 1 i.e. we are considering models of the form $N(\theta,1)$. In this case, the likelihood (for a single data point $x$) is (ignoring the constant) $L(\theta | x)=\exp(-(x-\theta)^2/2)$.

Suppose that we are actually interested in a function of the mean, call it $\eta=\tau(\theta)$. How to define the likelihood $L(\eta|x)$? If $\tau$ is invertible then we just define $L(\eta|x)$ to be $L(\theta=\tau^{-1}(\eta) | x)$ i.e. we set $\theta$ equal to the unique value corresponding to the chosen value of $\eta$. e.g. if $\tau(\theta)=2\theta$ then $L(\eta | x):=L(\theta=\frac{\eta}{2} |x)$.

What if $\tau$ is not invertible? e.g. $\tau(\theta)=\theta^2$. Should $L(\eta|x)$ be $L(\theta=+\sqrt{\eta} | x)$ or should it be defined as $L(\theta=-\sqrt{\eta} | x)$? These two values will usually be different, so the likelihood $L(\eta|x)$ is undefined. Hence Casella and Berger define the induced likelihood. With the chosen definition, it turns out that the invariance property (which is obvious when $\tau$ is invertible) still holds.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I think I have misunderstood as to what is happening. We are interested in a parameter of a distribution which satisfies a constraint. Say $\eta $<1. Then we define a function which satisfies the constraint, say $\eta = \tau(\theta) = exp(\theta) / ( exp(\theta)+1) $. Then, upto a constant, we will have $L(\eta) = exp(-(x-\eta)^2/2)$. Seems to me that I have inverted the role of the dependent and independent variable, but we are interested in the parameter satisfying a constraint ie. $\eta$ should be $\tau(\theta)$ and not the other way. Where do I misunderstand? $\endgroup$
    – user2338823
    Commented Sep 9, 2017 at 9:24
  • 1
    $\begingroup$ Not sure why the constraint is relevant here? The function you specified is invertible, so in this case $L(\eta)=L(\theta=\tau^{-1}(\eta))=\exp(-(x-\log(\frac{\eta}{1-\eta}))^2/2)$. $\endgroup$
    – S. Catterall
    Commented Sep 9, 2017 at 12:32
  • $\begingroup$ In your 1st comment, we do not want a function of the mean. Instead we want the mean to be a function of a parameter, so that it obeys the constraint implied by being a function of that parameter. That's where I disagree with you. $\endgroup$
    – user2338823
    Commented Sep 9, 2017 at 14:45
  • $\begingroup$ Ok, so it sounds like your set-up is a bit different from (the reverse of?) the one in my example above. You have a parameter $\eta$ constrained to lie between $0$ and $1$, and presumably an associated likelihood $L(\eta)$. Then you're defining a transformed parameter $\theta=\log(\frac{\eta}{1-\eta})$. The transformation is invertible, so your likelihood is just $L(\theta)=L(\eta=\frac{\exp(\theta)}{1+\exp(\theta)})$. $\endgroup$
    – S. Catterall
    Commented Sep 10, 2017 at 10:10
  • $\begingroup$ Suppose we have a different transformation. say $\eta= \tau(\theta) = \theta^2$. Then $L(-\theta) = L(\theta) = L(\eta) = L((+/-\theta)^2)$. Now would we need to define the induced likelihood as before ? I think all $\theta's$ which map to the same $\eta$ will have the same likelihood. Is that correct? I am doing this in context to state-space-model , page 28. Have I completely misunderstood what is happening? $\endgroup$
    – user2338823
    Commented Sep 11, 2017 at 4:17
1
$\begingroup$

I believe a lot of misunderstanding arises from notation used. The best explanation of what induced likelihood function is is in the original paper of Zhenna, 1966 (see 1). Induced likelihood function is one of ways to make $\tau(\theta)$ one-to-one when it is not one-to-one initially. Several methods of achieving this exist, see "On invariance and maximum likelihood estimation" [2]. This $L_{1}^*(\eta|x) = sup_{\theta|\tau(\theta) = \eta} L(\theta|x) \tag{1}$ tells to map all values of $\tau(\theta)$ to corresponding maximum value of $\theta$. An example. Suppose $\tau(\theta)=\sin(\theta) , f(\theta)=2*\theta+5, \theta \in [0,100]$. Obviously, $sup f(\theta) = 205$ The transformation is not one-to-one. Supremum value is attained at $\theta=100$, but many values of $x$ satisfy $sin(x)=sin(100)$. All such values of $x$ are mapped by induced likelihood function to single value, the supremum value, 205 in this example. Thus it is assured that both functions likelihood and induced likelihood are maximized so invariance is preserved.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Wooki Woki, many thanks for your reply. I am a little confused though. In your example , $L(\eta) = 2 * \eta + 5$ where $\eta$ is in [0,100]. When I parametrize $\eta = \tau(\theta) = sin(\theta) $ I will never have an answer bigger than 7 since sin can be no more than 1. Can you please tell me where I misunderstand? $\endgroup$
    – user2338823
    Commented Apr 22, 2019 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.