Is entropy conserved under invertible mappings?

Question

Suppose for a random variable $ X\colon \Omega \to E $, I have an invertible mapping $ Z = f(X) $.

Is the Shannon Entropy for each variable equivalent?

$$ H(X) = H(Z) $$

If not, can anything relationship between the entropies be said in general (eg. proportionality, additive constant, etc.)?

Answer 1

Is the Shannon Entropy for each variable equivalent?

Yes. An intuitive way of seeing why this is so is that (deterministic) transformations cannot increase entropy.

Specifically, note that $$H(X) \geq H(f(X)) = H(Z),$$ because, by the chain rule of entropy:

$$ H(X, f(X)) = H(X) + H(f(X) | X) = H(X), $$

but also

$$ H(X, f(X)) = H(f(X)) + H(X | f(X)) \geq H(f(X)) . $$

However, letting $g = f^{-1}$ (which exists, as $f$ is invertible), also $$H(Z) \geq H(g(X)) = H(X).$$

Answer 2

Actually I would disagree with Nick or Ami. Although I am not familiar with the chain rules of entropy, we can use a simple counter example. Suppose $X\sim N(0,\sigma^2)\in \mathbb{R}$. Then its entropy is given by (derivation given here) $$ H=\frac{1}{2}\log(2\pi)+\log(\sigma)+\frac{1}{2} $$ The entropy is dependent on $\sigma$, which can be modified by scaling. e.g. $Y=2X$ implies $\sigma_Y=2\sigma_X$, which yields $H(Y)-H(X)=\log 2\neq 0$

Edit: I realised that depending on whether X is discrete or continuous, the answer could be different. The first link by R Carnell illustrates this difference.

Answer 3

The answer given by Ami is correct, I will provide an intuitive explanation of why $H(X|Z) = H(Z|X) = 0$:

1. $f(\cdot)$ is a deterministic mapping from $X$ to $Z$, so when you observe $X$, you have zero uncertainty to infer about $Z$, thus $H(Z|X) = 0$.
2. Note that $Z=f(X)$ is invertible, so when you observe $Z$, you also have zero uncertainty to infer about $X$, that's to say, $H(X|Z)=0$.

Combining 1 and 2, it is clear that $H(X)=H(X,Z)-H(Z|X)=H(X,Z)$ and $H(Z)=H(X,Z)-H(X|Z)=H(X,Z)$, so $H(X)$ and $H(Z)$ are exactly the same.