7
$\begingroup$

There is something that is confusing me about max-likelihood estimators. Suppose my I have some data and the likelihood under a parameter $\mu$ is

$$ L(D|\mu) = e^{-(.7-\mu)^2} $$

which is recognizable as the likelihood of Gaussian upto scaling. Now my max likelihood estimator will give me $\mu=.7$.

Now suppose I didn't know that and instead was working with a parameter $t$ such that $\mu=\sin(t)$. Also suppose all this were numerical and so I wouldn't immediately see how silly the following likelihood looks like

$$ L(D|t) = e^{-(.7-\sin(t))^2} $$

Now I would solve for the max likelihood and get additional solutions. To help see this I plot it below.

enter image description here

So from this point of view max-likelihood seems like a silly thing to do as it is not re-parametrization invariant. What am I missing?

Note that a Bayesian analysis would naturally take care of this since the likelihoods would always come with a measure

$$ L(D|\mu) P(\mu) d\mu = L(D|\mu(t)) P(\mu(t)) \frac{d\mu}{dt} dt $$

Added part after responses and comments (added on 3/16/2018)

I realized later that my example above is not a good one because the two maxima in $t_1,t_2$ correspond to $.7=\sin(t_1)=\sin(t_2)$. So they are identifying the same point. I have kept the above for the discussion and responses below to make sense. However, I think the following is a better example of the issue I am trying to figure out.

Take

$$ L(D|\mu) = e^{-(a-\mu)^2} $$

Now suppose I reparameterize $\mu=\mu(t)$ then do a max-likelihood with respect to $t$ I get

$$ \frac{\partial L}{\partial t} = \frac{\partial L}{\partial \mu} \frac{\partial \mu}{\partial t} $$

If I want a maxima at a location other than from the one I get from maximizing with respect to $\mu$ I require

$$ \frac{\partial L}{\partial \mu} \ne 0 $$

and

$$ \frac{\partial \mu}{\partial t} =0, \qquad \frac{\partial L}{\partial \mu} \frac{\partial^2 \mu}{\partial t^2} < 0 $$

Thus, I can take a simple example

$$ \mu = b - (a-b)t^2+t^3 $$

I plot the results below. We can clearly see that $\mu=a$ is the global maxima (and only one when maximizing with respect to $\mu$) but we also have another local maxima at $t=0$ when we maximizing with respect to $t$.

enter image description here

Note the map $\mu(t)$ is not bijective but I do not see why it has to be. Also, at least in this example, the global maxima will always be the one at $\mu=a$ but from a frequentist point of view wouldn't I be oblidged to take some kind of weighted average of 1/1.6 of $\mu=a$ and .6/1.6 of $\mu=b$ (that corresponds to $t=0$) if I completely worked in the $t$ space?

$\endgroup$
  • 11
    $\begingroup$ On the contrary, the solution is invariant. The correct formulation is that all the values $t$ which minimize $L(\mu(t))$ correspond to values of $\mu$ that minimize $L(\mu)$--which ought to be obvious from the notation alone. For this result to hold, it doesn't matter whether $\mu$ is invertible, one-to-one, continuous, or anything else, because in the end all we're discussing is how to name the distribution(s) for which the likelihood is largest. "A rose by any other name smells just as sweet." $\endgroup$ – whuber Mar 15 '18 at 20:38
  • $\begingroup$ I had to edit as my example was not good. New example is $L=e^{-(a-\mu)^2}$ and $\mu=b-(a-b) t^2 + t^3$. This gives an additional 'local' maxima at $t=0$. The likelihood is not bimodal so shouldn't one take weighted average? If so this would make the solution not invariant. $\endgroup$ – Borun Chowdhury Mar 16 '18 at 9:26
  • $\begingroup$ @whuber I do agree that since likelihood is a scalar, the global maxima is invariant under reparameterization just like a rose with another name smells just as sweet. I was talking more about the possibility of generating several local maxima and then justifying not taking the weighted average. $\endgroup$ – Borun Chowdhury Mar 16 '18 at 11:33
  • $\begingroup$ No average would necessarily make any sense at all, because in the end you are describing distributions rather than numbers. In your many-to-one transformation, you ought to be "averaging" the same distribution with itself--because all the maxima correspond to the same distribution--but the average of the numerical "names" you assigned these distributions would be meaningless. $\endgroup$ – whuber Mar 16 '18 at 13:41
15
$\begingroup$

Looking at your graph, it appears that $\hat{t} \in \{0.7753975, 2.346194\}$ is a pretty reasonable guess at the MLE(s) of $t$. Running those values through the $\sin$ function to get back to $\mu$ results in $\hat{\mu} = \{0.7, 0.7\}$ or $0.7$, just as it should. So, there is no disagreement between the MLE of $\mu$ and the MLE(s) of $t$.

What is happening is that you have created a map from $\mu \to t$ that is not 1-1. In this case, the true value of $\mu$ maps to multiple values of $t$, so not surprisingly you will have multiple maxima when working with $t$. Note, however, that this would be the same if you were doing a Bayesian analysis, unless your prior restricted $t$ to the interval $[-\pi/2, \pi/2)$ or some such. If you did so, for comparability, you should restrict the range of the MLE of $t$ to the same range, in which case you won't get multiple maxima for the likelihood function any more.

ETA: In retrospect I focused too much on explanation-by-example and not enough on the underlying principle. One can hardly do better than @whuber's comment in response to the OP in this regard.

In general, if you have a parameter $\theta$ and an associated MLE $\hat{\theta}$, and you construct a function $\theta = f(t)$, you've effectively created an alternate parameter $t$. The MLE of $t$, label it $\hat{t}$, will be those values of $t$ such that $f(t) = \hat{\theta}$, i.e., $f(\hat{t}) = \hat{\theta}$.

$\endgroup$
  • $\begingroup$ I agree that my example is not quite what I expected it to be. I realized it on the way back home. A better example is $\mu=b-t^2+t^3$. Here we get maxima for $t$ that do not map to $\mu=.7$ (depending on $b$). Nevertheless, it is also not bijective. $\endgroup$ – Borun Chowdhury Mar 15 '18 at 20:39
  • $\begingroup$ I don't see why the parametrization has to be bijective. In fact, I am not asking which parameterizations can be done to give the same answer, I am asking why maximum likelihood is used when it is not re-parameterization invariant. $\endgroup$ – Borun Chowdhury Mar 15 '18 at 20:43
  • $\begingroup$ Some of my best thinking is done in rush hour traffic... Can you come up with a $\mu$, $b$ and $t$ for which $\mu = b - t^2 + t^3$ results in different values for the likelihood function when you plug $\mu$ in than when you plug $b - t^2 + t^3$ in in place of $\mu$? I'm guessing not... see @whuber's comment above. $\endgroup$ – jbowman Mar 15 '18 at 21:18
  • $\begingroup$ My point about the not 1-1 nature of your function isn't that it doesn't work at all, it's that that is what is responsible for the multimodality of the likelihood function for $t$ (well that and that the function isn't 1-1 at $\mu$, which is clearly more restrictive.) $\endgroup$ – jbowman Mar 15 '18 at 21:25
  • $\begingroup$ I edited to include the aforementioned example. I took $\mu= b- (a-b) t^2 + t^3$. Then as long as $a \ne b$ there is an additional 'local' maxima at $t=0 (\mu=b)$. Even though its a local maxima, since its height is comparable, the max-likelihood should be a weighted average of $\mu=a,b$ (I am guessing that is what is done for bimodal max-likelihood). $\endgroup$ – Borun Chowdhury Mar 16 '18 at 9:23
1
$\begingroup$

As my previous answer wasn't completely clear about the need for bijectivness or not (one could argue my answer was just plain wrong). I did some research about the whole reparametrizing thing and here is what I found out. Both @whuber and @jbowman touch upon some of the same things.

Theory

So, in theory, the maximum likelihood estimator $\hat{\theta}$ of the likelihood function $L\left(\theta\right)$, is invariant to re-parametrization. So, say you have some known function $g$, which re-parametrizes $\theta$ into $\lambda=g(\theta)$ (where the dimensions of $\theta$ and $\lambda$ are not necessarily the same). Then two facts hold true,

  • Maximizing $L\left(\theta\right)$ wrt. $\theta$, that is, finding the MLE, $\hat{\theta}$, and then reparametrizing it, $g(\hat{\theta})$, yields the MLE of $\hat{\lambda}$. In short, $\hat{\lambda}=g\left(\hat{\theta}\right)$.
  • Further, if $g$ has an inverse, maximizing $L\left(g^{-1}(\lambda)\right)$ wrt. $\lambda$, that is, finding the MLE $\hat{\lambda}$ yields the same maximum as $\hat{\theta}$. So the MLE of $\theta$ is $\hat{\theta}=g^{-1}\left(\hat{\lambda}\right)$.

Splitting the invariance in these two sub-cases can seem a bit artificial, but I find it useful since they represent two different use-cases of re-parametrization.

In practice

The first use-case is where you somehow can identify the MLE for some parameter, but you actually need a certain transformation of that variable. For example you have an estimator, $\hat{\sigma},$ for the parameter $\sigma$ in the normal distribution, but you are actually interested in the MLE for the variance $\sigma^{2}$. Then you can use the invariance principle and simply square the $\sigma$-MLE, $\hat{\sigma^{2}}=(\hat{\sigma})^{2}$.

An example for the second use-case, is that you have a numerical algorithm, like gradient descent or Newton-Raphson, to maximize the likelihood function. Say, you want to estimate the parameter $\sigma^{2}$ from a normal distribution. The parameter is strictly positive by definition, but the numerical procedure doesn't allow you to make constraints. Well, you can use the invariance property to set $\sigma^{2}=\exp(\lambda)$ and let the algorithm vary $\lambda$ instead of $\sigma^{2}$, this way ensuring that $\sigma^{2}$ stays positive. The exponential is bijective, but this is not strictly required. We could have used $\sigma^{2}=\lambda^{2}$ instead, which is not bijective. But using a bijection is more practical, since we can go from $\sigma^{2}$ to $\lambda$ and back in a unique fashion.

The formalities

To define the MLE of $\lambda$ more formally we need to define what is called the profile likelihood function as,

$$L^{\ast}(\lambda)=\sup_{\theta\vert\lambda=g\left(\theta\right)}L\left(\theta\right).$$

So, for a given $\lambda$-value the profile likelihood value, is the supremum over all $\theta$'s which ensure that $g\left(\theta\right)$ equals $\lambda$.

With the profile likelihood defined we can then define the MLE for $\lambda$, denoted $\hat{\lambda}$, as the value which maximizes $L^{\ast}\left(\lambda\right)$.

With these definitions in place the invariance of re-parametrization boils down to,

$$ L^{\ast}\left(\hat{\lambda}\right)=L\left(\hat{\theta}\right) $$

which can be proved by,

$$L^{\ast}\left(\hat{\lambda}\right)=\max_{\lambda}L^{\ast}\left(\lambda\right)=\max_{\lambda}\sup_{\theta\vert\lambda=g\left(\theta\right)}L\left(\theta\right)=\sup_{\theta}L\left(\theta\right)=\max_{\theta}L\left(\theta\right)$$

where I have assumed that $L\left(\theta\right)$ has a maximum.

If the re-parametrization is a bijection i.e. it is invertible, then $L^{\ast}\left(\lambda\right)$ is simply $L(g(\theta))$ since each $\theta$ uniquely maps to a $\lambda$, and hence the supremum over ``all'' $\theta$'s just collapses to the unique $L(\theta)$. So, we get that,

\begin{align*} L^{\ast}\left(\lambda\right) & =L\left(g\left(\theta\right)\right)\\ L^{\ast}\left(g^{-1}(\lambda)\right) & =L\left(\theta\right) \end{align*} and hence,

$$\hat{\theta}=g^{-1}\left(\hat{\lambda}\right).$$

References:

Invariance property of MLE: what is the MLE of $\theta^2$ of normal, $\bar{X}^2$?

http://www.stats.ox.ac.uk/~dlunn/b8_02/b8pdf_6.pdf

http://www.stat.unc.edu/faculty/cji/lecture7.pdf

https://en.wikipedia.org/wiki/Maximum_likelihood_estimation#Functional_invariance

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.