How does entropy depend on location and scale?

Question

The entropy of a continuous distribution with density function $f$ is defined to be the negative of the expectation of $\log(f),$ and therefore equals

$$H_f = -\int_{-\infty}^{\infty} \log(f(x)) f(x)\mathrm{d}x.$$

We also say that any random variable $X$ whose distribution has density $f$ has entropy $H_f.$ (This integral is well-defined even when $f$ has zeros, because $\log(f(x))f(x)$ can be taken to equal zero at such values.)

When $X$ and $Y$ are random variables for which $Y = X+\mu$ ($\mu$ is a constant), $Y$ is said to be a version of $X$ shifted by $\mu.$ Similarly, when $Y = X\sigma$ ($\sigma$ is a positive constant), $Y$ is said to be a version of $X$ scaled by $\sigma.$ Combining a scale with a shift gives $Y=X\sigma + \mu.$

These relations occur frequently. For instance, changing the units of measurement of $X$ shifts and scales it.

How is the entropy of $Y = X\sigma + \mu$ related to that of $X?$

Answer 1

Since the probability element of $X$ is $f(x)\mathrm{d}x,$ the change of variable $y = x\sigma + \mu$ is equivalent to $x = (y-\mu)/\sigma,$ whence from

$$f(x)\mathrm{d}x = f\left(\frac{y-\mu}{\sigma}\right)\mathrm{d}\left(\frac{y-\mu}{\color{red}\sigma}\right) = \frac{1}{\color{red}\sigma} f\left(\frac{y-\mu}{\sigma}\right) \mathrm{d}y$$

it follows that the density of $Y$ is

$$f_Y(y) = \frac{1}{\color{red}\sigma}f\left(\frac{y-\mu}{\sigma}\right).$$

(Keep an eye on that $1/\color{red}\sigma$ factor in the subsequent derivation--and remember it appears here because $y$ is a uniformly rescaled version of $x;$ that is, $\mathrm d x = \mathrm d y / \color{red}\sigma.$)

According to the definition, the (differential) entropy of $Y$ is

$$H(Y) = -\int_{-\infty}^{\infty} \log\left(\frac{1}{\color{red}\sigma}f\left(\frac{y-\mu}{\sigma}\right)\right) \frac{1}{\color{red}\sigma}f\left(\frac{y-\mu}{\sigma}\right) \mathrm{d}y.$$

Upon changing the variable back to $x = (y-\mu)/\sigma$ this becomes

$$\eqalign{ H(Y) &= -\int_{-\infty}^{\infty} \log\left(\frac{1}{\color{red}\sigma}f\left(x\right)\right) f\left(x\right) \mathrm{d}x \\ &= -\int_{-\infty}^{\infty} \left(\log\left(\frac{1}{\color{red}\sigma}\right) + \log\left(f\left(x\right)\right)\right) f\left(x\right) \mathrm{d}x \\ &= \log\left(\color{red}\sigma\right) \int_{-\infty}^{\infty} f(x) \mathrm{d}x -\int_{-\infty}^{\infty} \log\left(f\left(x\right)\right) f\left(x\right) \mathrm{d}x \\ &= \log(\color{red}\sigma) + H_f. }$$

These calculations used basic properties of the logarithm, the linearity of integration, and the fact that $f(x)\mathrm{d}x$ integrates to unity (the Law of Total Probability).

The conclusion is

The (differential) entropy of $Y = X\sigma + \mu$ is the entropy of $X$ plus $\log(\sigma).$

In words, shifting a random variable does not change its entropy (we may think of the entropy as depending on the values of the probability density, but not on where those values occur), while scaling a continuous variable (which, for $\sigma \ge 1$ "stretches" or "smears" it out) increases its entropy by $\log(\sigma).$ This supports the intuition that high-entropy distributions are "more spread out" than low-entropy distributions.

---

As a consequence of this result, we are free to choose convenient values of $\mu$ and $\sigma$ when computing the entropy of any distribution. For example, the entropy of a Normal$(\mu,\sigma)$ distribution can be found by setting $\mu=0$ and $\sigma=1.$ The logarithm of the density in this case is

$$\log(f(x)) = -\frac{1}{2}\log(2\pi) - x^2/2,$$

whence

$$H = -E\left[-\frac{1}{2}\log(2\pi) - X^2/2\right] = \frac{1}{2}\log(2\pi) + \frac{1}{2}.$$

Consequently the entropy of a Normal$(\mu,\sigma)$ distribution is obtained simply by adding $\log\sigma$ to this result, giving

$$H = \frac{1}{2}\log(2\pi) + \frac{1}{2} + \log(\sigma) = \frac{1}{2}\log\left(2\pi\,e\,\sigma^2\right)$$

as reported by Wikipedia.