1
$\begingroup$

Given a univariate sample $\vec X = X_1, ..., X_n$ with standard deviation 1 and a strictly monotone transformation $t: R \to R$ with the property that the standard deviation of $t(\vec X)$ is also 1 (where $t(\vec X)$ is $t$ applied to each $X_i$). If one know fits a normal distribution to $\vec X$ and $t(\vec X)$, one observes the following fact: the likelihood at the respective MLEs of $\vec X$ and $t(\vec X)$ is the same. The reason for that is, that the standard deviation is the MLE for the $\sigma$ parameter of the normal distribution, so just the mean changes, which does not change the likelihood.

In some computations with multivariate data I observed the same fact, using transformations such that $\det(cov(t(\vec X)))=1)$. But I did not see a straightforward proof for that! Note that in the multivariate case, the MLEs for the variance parameters do change, even if the above determinant is always 1.

I am sure I am not the first to notice, but my (somewhat limited) literature on the multivariate normal distribution does not give me a clue how to prove the above statement, i.e. that the multivariate normal likelihood does only depend on the determinant of the $cov(t(\vec X))$, regardless of choice of $t$.

A proof boils down to looking at the terms of the form $x'\Sigma x$ for $\det \Sigma = 1$.

Do you have more clue?

Thanks, Philipp

$\endgroup$
2
$\begingroup$

This is due to the invariance property of MLEs. It is an elementary exercise in calculus to show that if $E_{MLE}$ maximises the function $F(E)$ then any monotonic function of $E$, say $g=g(E)$ will also be maximised at $g_{MLE}=g(E_{MLE})$.

$F(E)=F[g^{-1}(E)]$ and therefore the maximum occurs at $F_{MAX}=F(E_{MLE})=F[g^{-1}(E_{MLE})]$, which means that $g_{MLE}=g(E_{MLE})$.

If $g(.)$ were not monotone (1-to-1 so that $g^{-1}(.)$ is well defined), then the above argument would not hold.

EDIT: I think this is why the MLE's change for the determinant transformation, perhaps the $t(.)$ function is not a 1-to-1 function. It is all about the functional properties of your $t(.)$ function (whether it is 1-to-1 or not).

EDIT#2: After some discussion, the transformation $t(.)$ may be 1-to-1, but if the inverse transformation $t^{-1}(.)$ doesn't have a Jacobian of 1, then the value of the likelihood at its maximum will be different. Fixing the variance to be 1 may one way of achieving this property.

The class of transformations $t(X)$ has a approximate (exact for normals?) covariance of $$cov[t(X)]\approx (\frac{\partial t(x)}{\partial t})_{x=X_{MLE}}^T cov[X](\frac{\partial t(x)}{\partial t})_{x=X_{MLE}}$$.
Fixing the determinant to 1 fixes (approximately) $$|\frac{\partial t(x)}{\partial t}|^{-2}_{x=X_{MLE}}\approx |cov[X]|$$ Now if $t(.)$ is invertible, the jacobian of the inverse transformation is just the inverse of the jacobian. SO we have: $$|\frac{\partial t(x)}{\partial t}|^{-1}_{x=X_{MLE}}=|\frac{\partial t^{-1}(x)}{\partial t}|_{x=X_{MLE}}\approx |cov[X]|^{\frac{1}{2}}$$

Now in the normal distribution, the likelihood evaluated at the MLE, the exponential part just becomes 1. So the maximised likelihood for X is given by

$$f_X(x_{MLE}) = \frac{1}{(2\pi)^{\frac{N}{2}} |cov[X]|^{\frac{1}{2}}}$$

using the Jacobian rule for transforming to $T=t(X)$ gives

$$f_T(t_{MLE}) = \frac{|\frac{\partial t^{-1}(x)}{\partial t}|_{x=X_{MLE}}}{(2\pi)^{\frac{N}{2}} |cov(X)|} \approx\frac{|cov[X]|^{\frac{1}{2}}}{(2\pi)^{\frac{N}{2}} |cov[X]|^{\frac{1}{2}}} = (2\pi)^{-\frac{N}{2}}$$

Which is a constant, as your analysis shows (you could even check my calculations, by comparing the likelihood you get with $(2\pi)^{-\frac{N}{2}}$, and see if it is right). I would imagine that either the approximation is exact for normals, or exact for certain transformations $t(.)$, or becomes exact as $N\rightarrow\infty$, or the approximation error is in decimal places which you checked (or can't calculate), OR the computer you use to get the MLEs of $T$ uses the same approximation technique as I have (called the delta method).

$\endgroup$
  • $\begingroup$ I am aware of te invariance property. My question refered to the fact, that the likelihood at the MLEs does not change. If you for example consider a trivial transform like $x \mapsto 0.1 * x$, then this transform will reduce variance because all the points move closer together. Hence the likelihood will increase, because of the factors of the form $1/ \sqrt(2 \pi \sigma^2)$. $\endgroup$ – Philipp Jan 26 '11 at 0:16
  • 1
    $\begingroup$ Nope, the likelihood will remain exactly the same because a jacobian factor will come out of the transformation to make them equal (dx to 0.1 dx) $\endgroup$ – probabilityislogic Jan 26 '11 at 1:09
  • $\begingroup$ I am not talking about the likelihood of $x$ under the new distribution obtained by plugging in the transformation, but about the likelihood of $t(x)$! That is something different. $\endgroup$ – Philipp Jan 26 '11 at 15:59
  • $\begingroup$ Okay, here's how it goes, start with CDF $F_{T}(t_0)=Pr(T<t_0)$, then we know $T=t(X)$ sub this in to get $F_{T}(t_0)=Pr(t(X)<t_0)=Pr(X<t^{-1}(t_0))=F_{X}(t^{-1}(t_0))$ To get the pdf you have to differentiate *with respect to $t_0$, which leaves us with $f_{T}(t_0)=f_{X}(t^{-1}(t_0)) \times |\frac{\partial t^{-1}(t_0)}{\partial t_0}|$. It is this Jacobian I was taking about, which makes the "maximum" of the likelihood equal, whenever $t(x)$ is a 1-to-1 function. $\endgroup$ – probabilityislogic Jan 26 '11 at 23:29
  • $\begingroup$ ...In your example we have $T=0.1X$ so $t(x)=0.1x$ and $t^{-1}(x)=10x$ with derivative 10, and $f_{T}(t)=f_{X}(10t)10$ Which, in the normal case you will have $f_{T}(t)=\frac{10}{\sigma \sqrt{2\pi}}exp(-\frac{1}{2\sigma^2}(10t-\mu)^2)$. So it would appear you're right. My thinking now, is that in fixing the variance, you are also fixing the Jacobian to be equal to 1 for certain functions $t(.)$ (i.e. functions who's inverse have derivative 1). This may explain why it doesn't always work. $\endgroup$ – probabilityislogic Jan 26 '11 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.