That is, say I have a distribution with parameter $\theta$.

If I re-write it with a parameter $a$ such that $a^3=\theta$, is it possible that doing maximum likelihood estimation on a will yield an estimate $\hat a$ such that $\hat a^3 \neq \hat \theta$?

Could it be the case for another function different than $x^3$?

If so, what are some criteria to choose a parametrization?


The Invariance Property of Maximum Likelihood Estimators (MLEs) says, if $\hat{\theta}$ is the MLE of $\theta$, then for any function $\tau(\theta)$ the MLE of $\tau(\theta)$ is $\tau(\hat{\theta})$.

So, if you define $a^3=\theta$, once you obtained your MLE for $\theta$, $\hat{\theta}$, you can apply the inverse function by taking the cubed root of $\hat{\theta}$ and obtain the MLE of $a$ (i.e. $\hat{a}=\hat{\theta}^{1\over{3}}$)


I've added the proof mentioned by Thomas Lumley in the comments:

Let $\hat{\eta}$ denote the value that maximizes $L^*(\eta|\textbf{x})$. We must show that $L^*(\hat{\eta}|\textbf{x})$=$L^*(\tau(\hat{\theta})|\textbf{x})$. The maxima of $L$ and $L^*$ coincide, so we have

\begin{eqnarray*} L^{*}(\hat{\eta}|\textbf{x}) & = & \underset{\eta}{\text{sup}}\underset{\{\theta:\tau(\theta)=\eta\}}{\text{sup}}\,L(\theta|\textbf{x})\\ & = & \underset{\theta}{\text{sup}}L(\theta|\textbf{x})\\ & = & L(\hat{\theta}|\textbf{x}), \end{eqnarray*}

The first and third equalities hold by definition of $L^{*}$ and $\hat{\theta}$ respectively, and the second equality holds because the iterated maximization is equal to the unconditional maximization over $\theta$, obtained at $\hat{\theta}$. Further,

\begin{eqnarray*} L(\hat{\theta}|\textbf{x}) & = & \underset{\{\theta:\tau(\theta)=\tau(\hat{\theta})\}}{\text{sup}}L(\theta|\textbf{x})\\ & = & L^{*}\left[\tau(\hat{\theta})|\textbf{x}\right]. \end{eqnarray*}

Hence, the string of equalities shows that $L^{*}(\hat{\eta}|\textbf{x})=L^{*}\left[\tau(\hat{\theta})|\textbf{x}\right]$ and that $\tau(\hat{\theta})$ is the MLE of $\tau(\theta)$. $\blacksquare$

  • 2
    $\begingroup$ Perhaps worth noting that the equivariance property holds even if the transformation isn't invertible. $\endgroup$ – Thomas Lumley Aug 17 '20 at 0:40
  • $\begingroup$ Good point, @ThomasLumley. I don't think I made that clear and, in fact, may have obfuscated that point with my example (or the OP's example). $\endgroup$ – StatsStudent Aug 17 '20 at 0:45
  • $\begingroup$ Wow. Very cool (and a bit surprising!). No conditions on tau? Do you happen to have a good statement of the theorem $\endgroup$ – josinalvo Aug 17 '20 at 2:58
  • 3
    $\begingroup$ The proof is much more straightforward for invertible transformations. This question math.stackexchange.com/questions/3246587/… gives a reference to the text by Casella and Berger for the general result. $\endgroup$ – Thomas Lumley Aug 17 '20 at 5:12
  • 2
    $\begingroup$ @ThomasLumley, I've added proof under the Update heading in case the OP does not have access to the book. $\endgroup$ – StatsStudent Aug 17 '20 at 5:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.