4
$\begingroup$

I know that maximum likelihood estimates are invariant to re-parametrization (https://stats.stackexchange.com/a/335368/267430). Is the MLE also invariant to rearrangements of the constants and parameters in the statistical model? Is this somehow just a case of re-parametrization that I am not seeing? For example, consider the model:

$\text{Model 1: }X \sim N(0, \sigma^2)$

I think this is equivalent to:

$\text{Model 2: }X/2 \sim N(0, \sigma^2/4)$

In other words, these are just two ways of writing down the exact same model. Yet, when you work out the log-likelihood of an observed data $x$ under each model, you get different results:

$\mathcal{L}_1 = -\frac{1}{2} \log(2\pi)-\frac{1}{2}\log(\sigma^2)-\frac{x^2}{2\sigma^2}$

$\mathcal{L}_2 = -\frac{1}{2} \log(2\pi)-\frac{1}{2}\log(\sigma^2/4)-\frac{(x/2)^2}{2\sigma^2/4}$

which are not equal, (although MLE estimates of $\sigma$ are equal).

In general (not only for this example, but for any similar situation where you are moving constants or parameters from the left-hand side to the right-hand side of the distributional assumptions), what is the relationship between $\mathcal{L}_1$ and $\mathcal{L}_2$? When doing maximum likelihood estimation for parameters in a model, how can you prove that the results don't change based on rearrangements in how the model is written down?

$\endgroup$
3
  • $\begingroup$ Your expressions ignore the implicit differential element $\mathrm{d}x.$ When you include that explicitly, the likelihoods are equal. $\endgroup$
    – whuber
    Dec 3, 2019 at 15:33
  • $\begingroup$ Thank you @whuber . If you have the chance, could you point me in the direction to learn more about the implicit differential element in a likelihood? I have to admit I've never heard of it, and I think I'm using the wrong search terms on Google/Wikipedia right now. $\endgroup$
    – Mon2
    Dec 3, 2019 at 16:25
  • $\begingroup$ The thread at stats.stackexchange.com/questions/248476 sheds considerable light on this issue. I have posted several answers discussing the role of probability elements in statistical theory, so perhaps some of those would be helpful too: here's the search $\endgroup$
    – whuber
    Dec 3, 2019 at 16:39

1 Answer 1

2
$\begingroup$

Your two log-likelihoods are equivalent, see What does "likelihood is only defined up to a multiplicative constant of proportionality" mean in practice?. Another way of saying that is that log-likelihoods are only defined up to an additive constant. Here constant means a term not depending on unknown parameters (in your example $\sigma^2$.)

By using properties of logarithms we have $$ \mathcal{L}_2 = -\frac{1}{2} \log(2\pi)-\frac{1}{2}\log(\sigma^2)+\frac12 \log 4 -\frac{x^2/4}{2\sigma^2/4} $$ and now simplifying and comparing to $\mathcal{L}_1$ you can see the difference is an additive constant, so those two likelihood functions are equivalent.

Generalizing from the example: The transformation $x\mapsto x/2$ is a completely specified and invertible function. It does not depend on any unknown parameters. So calculating the density function of the transformed variable involves a Jacobian, which only depends on the transformation, and is completely known without dependence on unknown parameters. The Jacobian will enter the loglikelihood function based on the transformed data as an additive constant, so the above argument applies.

$\endgroup$
3
  • $\begingroup$ Can it be proved that this holds in general for any rearrangement in the way any model is written? Or would that be considered obvious/trivial $\endgroup$
    – Mon2
    Dec 3, 2019 at 15:52
  • $\begingroup$ In particular, why doesn't this argument seem to apply to the pair Model 1: $X \sim N(0, \sigma^2)$ and Model 2: $X/\sigma \sim N(0, 1)$? $\endgroup$
    – Mon2
    Dec 3, 2019 at 16:15
  • 2
    $\begingroup$ If $\sigma$ is known, it applies. Otherwise no. I will augment the answer. $\endgroup$ Dec 3, 2019 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.