# Mutual Information really invariant to invertible transformations?

"Estimating mutual information" [A Kraskov, H Stögbauer, P Grassberger - Physical Review E, 2004] states that

Mutual information is invariant under reparametrization of the marginal variables. If $X ′ = F(X)$ and $Y′ =G(Y)$ are homeomorphisms [ie. smooth uniquely invertible maps], then $$I(X,Y ) = I(X′,Y′)$$

This paper is also used on Wikipedia to justify the same claim.

But if that is true, doesn't that imply that if $F=G$ and $X=Y$ we get $$H(X) = I(X,X) = I(X',X') = I(F(X),F(X)) = H(F(X))$$ where $H$ is the differential entropy?

Wouldn't this be "proof" that differential entropy is also invariant wrt. such transformations (which it obviously isn't because e.g. for a constant $a$, $H(aX) = H(X) + \log|a| \neq H(X)$)?

In particular I'm wondering if $I(X,F(X))=H(X)=H(F(X))$ for all homeomorphisms $F$.

Can someone help me reduce my uncertainty?

PS: I'm talking about the continous case, ie. differential entropy and differential mutual information

• Two comments which might be helpful: (1) Did you see that a sketch of the proof of the desired result is given in the appendix of the paper? (2) Note that the distribution of $(X, X)$ is singular with respect to Lebesgue measure on $\mathbb R^2$ and so your manipulations are not particularly rigorous. While the self-information result is true for (discrete) entropy, more care is needed in the continuous case. – cardinal Feb 17 '13 at 21:22
• Thanks for the reply. (1) Yes, I saw the proof sketch in the appendix and it looked ok to me, but you are right with (2), maybe this singular case is special and needs special treatment. Unfortunately I'm not so fluent with measure theory, so I'd appreciate if someone could point me to something. Surely someone has considered $I(X,F(X))$ in the continuous case before, but I can't find anything about it. – user20948 Feb 18 '13 at 12:50