Maximum likelihood is not re-parametrization invariant. So how can one justify using it?

Question

There is something that is confusing me about max-likelihood estimators. Suppose my I have some data and the likelihood under a parameter $\mu$ is

$$ L(D|\mu) = e^{-(.7-\mu)^2} $$

which is recognizable as the likelihood of Gaussian upto scaling. Now my max likelihood estimator will give me $\mu=.7$.

Now suppose I didn't know that and instead was working with a parameter $t$ such that $\mu=\sin(t)$. Also suppose all this were numerical and so I wouldn't immediately see how silly the following likelihood looks like

$$ L(D|t) = e^{-(.7-\sin(t))^2} $$

Now I would solve for the max likelihood and get additional solutions. To help see this I plot it below.

So from this point of view max-likelihood seems like a silly thing to do as it is not re-parametrization invariant. What am I missing?

Note that a Bayesian analysis would naturally take care of this since the likelihoods would always come with a measure

$$ L(D|\mu) P(\mu) d\mu = L(D|\mu(t)) P(\mu(t)) \frac{d\mu}{dt} dt $$

Added part after responses and comments (added on 3/16/2018)

I realized later that my example above is not a good one because the two maxima in $t_1,t_2$ correspond to $.7=\sin(t_1)=\sin(t_2)$. So they are identifying the same point. I have kept the above for the discussion and responses below to make sense. However, I think the following is a better example of the issue I am trying to figure out.

Take

$$ L(D|\mu) = e^{-(a-\mu)^2} $$

Now suppose I reparameterize $\mu=\mu(t)$ then do a max-likelihood with respect to $t$ I get

$$ \frac{\partial L}{\partial t} = \frac{\partial L}{\partial \mu} \frac{\partial \mu}{\partial t} $$

If I want a maxima at a location other than from the one I get from maximizing with respect to $\mu$ I require

$$ \frac{\partial L}{\partial \mu} \ne 0 $$

and

$$ \frac{\partial \mu}{\partial t} =0, \qquad \frac{\partial L}{\partial \mu} \frac{\partial^2 \mu}{\partial t^2} < 0 $$

Thus, I can take a simple example

$$ \mu = b - (a-b)t^2+t^3 $$

I plot the results below. We can clearly see that $\mu=a$ is the global maxima (and only one when maximizing with respect to $\mu$) but we also have another local maxima at $t=0$ when we maximizing with respect to $t$.

Note the map $\mu(t)$ is not bijective but I do not see why it has to be. Also, at least in this example, the global maxima will always be the one at $\mu=a$ but from a frequentist point of view wouldn't I be oblidged to take some kind of weighted average of 1/1.6 of $\mu=a$ and .6/1.6 of $\mu=b$ (that corresponds to $t=0$) if I completely worked in the $t$ space?

Answer 1

Looking at your graph, it appears that $\hat{t} \in \{0.7753975, 2.346194\}$ is a pretty reasonable guess at the MLE(s) of $t$. Running those values through the $\sin$ function to get back to $\mu$ results in $\hat{\mu} = \{0.7, 0.7\}$ or $0.7$, just as it should. So, there is no disagreement between the MLE of $\mu$ and the MLE(s) of $t$.

What is happening is that you have created a map from $\mu \to t$ that is not 1-1. In this case, the true value of $\mu$ maps to multiple values of $t$, so not surprisingly you will have multiple maxima when working with $t$. Note, however, that this would be the same if you were doing a Bayesian analysis, unless your prior restricted $t$ to the interval $[-\pi/2, \pi/2)$ or some such. If you did so, for comparability, you should restrict the range of the MLE of $t$ to the same range, in which case you won't get multiple maxima for the likelihood function any more.

ETA: In retrospect I focused too much on explanation-by-example and not enough on the underlying principle. One can hardly do better than @whuber's comment in response to the OP in this regard.

In general, if you have a parameter $\theta$ and an associated MLE $\hat{\theta}$, and you construct a function $\theta = f(t)$, you've effectively created an alternate parameter $t$. The MLE of $t$, label it $\hat{t}$, will be those values of $t$ such that $f(t) = \hat{\theta}$, i.e., $f(\hat{t}) = \hat{\theta}$.

Answer 2

As my previous answer wasn't completely clear about the need for bijectivness or not (one could argue my answer was just plain wrong). I did some research about the whole reparametrizing thing and here is what I found out. Both @whuber and @jbowman touch upon some of the same things.

Theory

So, in theory, the maximum likelihood estimator $\hat{\theta}$ of the likelihood function $L\left(\theta\right)$, is invariant to re-parametrization. So, say you have some known function $g$, which re-parametrizes $\theta$ into $\lambda=g(\theta)$ (where the dimensions of $\theta$ and $\lambda$ are not necessarily the same). Then two facts hold true,

• Maximizing $L\left(\theta\right)$ wrt. $\theta$, that is, finding the MLE, $\hat{\theta}$, and then reparametrizing it, $g(\hat{\theta})$, yields the MLE of $\hat{\lambda}$. In short, $\hat{\lambda}=g\left(\hat{\theta}\right)$.
• Further, if $g$ has an inverse, maximizing $L\left(g^{-1}(\lambda)\right)$ wrt. $\lambda$, that is, finding the MLE $\hat{\lambda}$ yields the same maximum as $\hat{\theta}$. So the MLE of $\theta$ is $\hat{\theta}=g^{-1}\left(\hat{\lambda}\right)$.

Splitting the invariance in these two sub-cases can seem a bit artificial, but I find it useful since they represent two different use-cases of re-parametrization.

In practice

The first use-case is where you somehow can identify the MLE for some parameter, but you actually need a certain transformation of that variable. For example you have an estimator, $\hat{\sigma},$ for the parameter $\sigma$ in the normal distribution, but you are actually interested in the MLE for the variance $\sigma^{2}$. Then you can use the invariance principle and simply square the $\sigma$-MLE, $\hat{\sigma^{2}}=(\hat{\sigma})^{2}$.

An example for the second use-case, is that you have a numerical algorithm, like gradient descent or Newton-Raphson, to maximize the likelihood function. Say, you want to estimate the parameter $\sigma^{2}$ from a normal distribution. The parameter is strictly positive by definition, but the numerical procedure doesn't allow you to make constraints. Well, you can use the invariance property to set $\sigma^{2}=\exp(\lambda)$ and let the algorithm vary $\lambda$ instead of $\sigma^{2}$, this way ensuring that $\sigma^{2}$ stays positive. The exponential is bijective, but this is not strictly required. We could have used $\sigma^{2}=\lambda^{2}$ instead, which is not bijective. But using a bijection is more practical, since we can go from $\sigma^{2}$ to $\lambda$ and back in a unique fashion.

The formalities

To define the MLE of $\lambda$ more formally we need to define what is called the profile likelihood function as,

$$L^{\ast}(\lambda)=\sup_{\theta\vert\lambda=g\left(\theta\right)}L\left(\theta\right).$$

So, for a given $\lambda$-value the profile likelihood value, is the supremum over all $\theta$'s which ensure that $g\left(\theta\right)$ equals $\lambda$.

With the profile likelihood defined we can then define the MLE for $\lambda$, denoted $\hat{\lambda}$, as the value which maximizes $L^{\ast}\left(\lambda\right)$.

With these definitions in place the invariance of re-parametrization boils down to,

$$ L^{\ast}\left(\hat{\lambda}\right)=L\left(\hat{\theta}\right) $$

which can be proved by,

$$L^{\ast}\left(\hat{\lambda}\right)=\max_{\lambda}L^{\ast}\left(\lambda\right)=\max_{\lambda}\sup_{\theta\vert\lambda=g\left(\theta\right)}L\left(\theta\right)=\sup_{\theta}L\left(\theta\right)=\max_{\theta}L\left(\theta\right)$$

where I have assumed that $L\left(\theta\right)$ has a maximum.

If the re-parametrization is a bijection i.e. it is invertible, then $L^{\ast}\left(\lambda\right)$ is simply $L(g(\theta))$ since each $\theta$ uniquely maps to a $\lambda$, and hence the supremum over ``all'' $\theta$'s just collapses to the unique $L(\theta)$. So, we get that,

\begin{align*} L^{\ast}\left(\lambda\right) & =L\left(g\left(\theta\right)\right)\\ L^{\ast}\left(g^{-1}(\lambda)\right) & =L\left(\theta\right) \end{align*} and hence,

$$\hat{\theta}=g^{-1}\left(\hat{\lambda}\right).$$

References:

Invariance property of MLE: what is the MLE of $\theta^2$ of normal, $\bar{X}^2$?

http://www.stats.ox.ac.uk/~dlunn/b8_02/b8pdf_6.pdf

http://www.stat.unc.edu/faculty/cji/lecture7.pdf

https://en.wikipedia.org/wiki/Maximum_likelihood_estimation#Functional_invariance