Does the entropy of a random variable change under a linear transformation?

Question

Let $X$ be a random variable. If $Y=aX+b$, where $a,b \in \mathbb{R}$, is the entropy of $Y$ the same as the entropy of $X$?

Answer 1

This depends on whether $X$ is discrete or continuous.

If $X$ is discrete

Let $R_X$ be the range of $X$ such that: $$ R_X = \{x_1,x_2,...,x_N\} $$ And let $P(X=x_i)$ be the probability mass function of $X$. The entropy of $X$ is: $$ H(X) = -\sum_{i=1}^N P(X=x_i) \log{P(X=x_i)} $$ Next, let: $$ Y = aX+b $$ Where $a,b \in \mathbb{R}$. This means that the range of $Y$ is: $$ R_Y = \{y_1,y_2,...,y_N\} = \{ax_1+b,ax_2+b,...,ax_N+b\} $$ And the probability mass function of $Y$ is: $$ \begin{align} P(Y = y_i) &= P(aX+b = y_i) \\ &= P\left(X = \frac{y_i-b}{a}\right) \quad i=1,2,...,N \end{align} $$ So the entropy of $Y$ is: $$ \begin{align} H(Y) &= -\sum_{i=1}^N P(Y=y_i) \log{P(Y=y_i)} \\ &= -\sum_{i=1}^N P\left(X = \frac{y_i-b}{a}\right) \log{P\left(X = \frac{y_i-b}{a}\right)} \\ &= -\sum_{i=1}^N P(X=x_i) \log{P(X=x_i)} \\ &= H(X) \end{align} $$

The entropy of $Y$ is the same as the entropy of $X$ in the case that $X$ is discrete.

If $X$ is continuous

Let $X$ be a continuous random variable with probability density function $p_X(x)$. Also, let $h(X)$ be the differential entropy of $X$ such that: $$ h(X) = -\int_{x\in\mathcal{X}} p_X(x) \log{p_X(x)} \ dx $$ Where $\mathcal{X}$ is the support of $X$. Next, let: $$ Y = aX + b $$ Such that: $$ x = \frac{y-b}{a} $$ And the probability density function of $Y$ is: $$ \begin{align} p_Y(y) &= p_X\left(\frac{y-b}{a}\right) \cdot \left|\frac{dx}{dy}\right| \\ &= \frac{1}{|a|} p_X\left(\frac{y-b}{a}\right) \end{align} $$ The differential entropy of $Y$ is: $$ \begin{align} h(Y) &= -\int_{y\in\mathcal{Y}} p_Y(y) \log{p_Y(y)} \ dy \\ &= -\int_{y\in\mathcal{Y}} \frac{1}{|a|} p_X\left(\frac{y-b}{a}\right) \log{\frac{1}{|a|} p_X\left(\frac{y-b}{a}\right)} \ dy \end{align} $$ Substituting $y=ax+b$ yields: $$ h(Y) = h(X) + \log{|a|} $$ This equation is only valid for $a \neq 0$. So, in the case that $X$ is continuous, then the differential entropy of $Y$ is not the same as the differential entropy of $X$.

Answer 2

In general, if $X$ takes at most countable many values, then for any function $f$, the entropy $H(f(X))$ of the random variable $f(X)$ cannot be greater than the entropy $H(X)$ of $X$. In short: $H(X) \geq H(f(X))$ for any $f$.

This is due to the monotonicity of the logarithm, implying $$(a+b)\log(\frac{1}{a+b})=a\log(\frac{1}{a+b})+b\log(\frac{1}{a+b})\leq a\log(\frac{1}{a})+b\log(\frac{1}{a})$$ for any $a, b >0$. Now consider $A = \{x\in X(\Omega): \exists x'\in X(\Omega)\text{ with } x \neq x'\text{ and } f(x)=f(x')\}$ and notice that $$H(X) - H(f(X)) = \sum_{x\in A}P(X=x)\log\Big(\frac{1}{P(X=x)}\Big)-\sum_{y\in f(A)}P(f(X)=y)\log\Big(\frac{1}{P(f(X)=y)}\Big).$$