29
$\begingroup$

The entropy of a continuous distribution with density function $f$ is defined to be the negative of the expectation of $\log(f),$ and therefore equals

$$H_f = -\int_{-\infty}^{\infty} \log(f(x)) f(x)\mathrm{d}x.$$

We also say that any random variable $X$ whose distribution has density $f$ has entropy $H_f.$ (This integral is well-defined even when $f$ has zeros, because $\log(f(x))f(x)$ can be taken to equal zero at such values.)

When $X$ and $Y$ are random variables for which $Y = X+\mu$ ($\mu$ is a constant), $Y$ is said to be a version of $X$ shifted by $\mu.$ Similarly, when $Y = X\sigma$ ($\sigma$ is a positive constant), $Y$ is said to be a version of $X$ scaled by $\sigma.$ Combining a scale with a shift gives $Y=X\sigma + \mu.$

These relations occur frequently. For instance, changing the units of measurement of $X$ shifts and scales it.

How is the entropy of $Y = X\sigma + \mu$ related to that of $X?$

$\endgroup$

1 Answer 1

32
$\begingroup$

Since the probability element of $X$ is $f(x)\mathrm{d}x,$ the change of variable $y = x\sigma + \mu$ is equivalent to $x = (y-\mu)/\sigma,$ whence from

$$f(x)\mathrm{d}x = f\left(\frac{y-\mu}{\sigma}\right)\mathrm{d}\left(\frac{y-\mu}{\color{red}\sigma}\right) = \frac{1}{\color{red}\sigma} f\left(\frac{y-\mu}{\sigma}\right) \mathrm{d}y$$

it follows that the density of $Y$ is

$$f_Y(y) = \frac{1}{\color{red}\sigma}f\left(\frac{y-\mu}{\sigma}\right).$$

(Keep an eye on that $1/\color{red}\sigma$ factor in the subsequent derivation--and remember it appears here because $y$ is a uniformly rescaled version of $x;$ that is, $\mathrm d x = \mathrm d y / \color{red}\sigma.$)

According to the definition, the (differential) entropy of $Y$ is

$$H(Y) = -\int_{-\infty}^{\infty} \log\left(\frac{1}{\color{red}\sigma}f\left(\frac{y-\mu}{\sigma}\right)\right) \frac{1}{\color{red}\sigma}f\left(\frac{y-\mu}{\sigma}\right) \mathrm{d}y.$$

Upon changing the variable back to $x = (y-\mu)/\sigma$ this becomes

$$\eqalign{ H(Y) &= -\int_{-\infty}^{\infty} \log\left(\frac{1}{\color{red}\sigma}f\left(x\right)\right) f\left(x\right) \mathrm{d}x \\ &= -\int_{-\infty}^{\infty} \left(\log\left(\frac{1}{\color{red}\sigma}\right) + \log\left(f\left(x\right)\right)\right) f\left(x\right) \mathrm{d}x \\ &= \log\left(\color{red}\sigma\right) \int_{-\infty}^{\infty} f(x) \mathrm{d}x -\int_{-\infty}^{\infty} \log\left(f\left(x\right)\right) f\left(x\right) \mathrm{d}x \\ &= \log(\color{red}\sigma) + H_f. }$$

These calculations used basic properties of the logarithm, the linearity of integration, and the fact that $f(x)\mathrm{d}x$ integrates to unity (the Law of Total Probability).

The conclusion is

The (differential) entropy of $Y = X\sigma + \mu$ is the entropy of $X$ plus $\log(\sigma).$

In words, shifting a random variable does not change its entropy (we may think of the entropy as depending on the values of the probability density, but not on where those values occur), while scaling a continuous variable (which, for $\sigma \ge 1$ "stretches" or "smears" it out) increases its entropy by $\log(\sigma).$ This supports the intuition that high-entropy distributions are "more spread out" than low-entropy distributions.


As a consequence of this result, we are free to choose convenient values of $\mu$ and $\sigma$ when computing the entropy of any distribution. For example, the entropy of a Normal$(\mu,\sigma)$ distribution can be found by setting $\mu=0$ and $\sigma=1.$ The logarithm of the density in this case is

$$\log(f(x)) = -\frac{1}{2}\log(2\pi) - x^2/2,$$

whence

$$H = -E\left[-\frac{1}{2}\log(2\pi) - X^2/2\right] = \frac{1}{2}\log(2\pi) + \frac{1}{2}.$$

Consequently the entropy of a Normal$(\mu,\sigma)$ distribution is obtained simply by adding $\log\sigma$ to this result, giving

$$H = \frac{1}{2}\log(2\pi) + \frac{1}{2} + \log(\sigma) = \frac{1}{2}\log\left(2\pi\,e\,\sigma^2\right)$$

as reported by Wikipedia.

$\endgroup$
5
  • $\begingroup$ so there is an anyalytical link between the entropy and the moments of a distribution? stats.stackexchange.com/questions/484495/… $\endgroup$
    – develarist
    Commented Aug 26, 2020 at 22:07
  • 2
    $\begingroup$ @devel It's interesting that you ask that question in this context, because the present result gives a clear indication of how moments and entropy are related. In particular, because we can change the first moment of $X$ arbitrarily by adding $\mu$ without changing the entropy, there's no relationship whatever between $H$ and $\mu.$ With central moments we have the result that scaling (that is, multiplying the second central moment by $\sigma^2$) corresponds to shifting $H$ or, equivalently, scaling $\exp(H).$ Thus, relative values of $H$ give information about relative scales. $\endgroup$
    – whuber
    Commented Aug 27, 2020 at 11:58
  • 1
    $\begingroup$ This QA addresses a common, general, and fundamental problem in an explicit step-by-step way. $\endgroup$
    – Galen
    Commented Mar 16, 2022 at 0:47
  • $\begingroup$ @whuber - nice explanation. Just one point of clarification (no critique implied). Wikipedia has the entropy for a normal distribution as $ \frac {1}{2} (1+ \log (2 \sigma^2 \pi)) $ which differs from the above expression. Just making sure I'm using the right equation. $\endgroup$
    – Mari153
    Commented Jun 1, 2022 at 2:16
  • 1
    $\begingroup$ @Mari The two expressions are algebraically equivalent. $\endgroup$
    – whuber
    Commented Jun 1, 2022 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.