20
$\begingroup$

The entropy of a continuous distribution with density function $f$ is defined to be the negative of the expectation of $\log(f),$ and therefore equals

$$H_f = -\int_{-\infty}^{\infty} \log(f(x)) f(x)\mathrm{d}x.$$

We also say that any random variable $X$ whose distribution has density $f$ has entropy $H_f.$ (This integral is well-defined even when $f$ has zeros, because $\log(f(x))f(x)$ can be taken to equal zero at such values.)

When $X$ and $Y$ are random variables for which $Y = X+\mu$ ($\mu$ is a constant), $Y$ is said to be a version of $X$ shifted by $\mu.$ Similarly, when $Y = X\sigma$ ($\sigma$ is a positive constant), $Y$ is said to be a version of $X$ scaled by $\sigma.$ Combining a scale with a shift gives $Y=X\sigma + \mu.$

These relations occur frequently. For instance, changing the units of measurement of $X$ shifts and scales it.

How is the entropy of $Y = X\sigma + \mu$ related to that of $X?$

$\endgroup$
23
$\begingroup$

Since the probability element of $X$ is $f(x)\mathrm{d}x,$ the change of variable $y = x\sigma + \mu$ is equivalent to $x = (y-\mu)/\sigma,$ whence from

$$f(x)\mathrm{d}x = f\left(\frac{y-\mu}{\sigma}\right)\mathrm{d}\left(\frac{y-\mu}{\sigma}\right) = \frac{1}{\sigma} f\left(\frac{y-\mu}{\sigma}\right) \mathrm{d}y$$

it follows that the density of $Y$ is

$$f_Y(y) = \frac{1}{\sigma}f\left(\frac{y-\mu}{\sigma}\right).$$

Consequently the entropy of $Y$ is

$$H(Y) = -\int_{-\infty}^{\infty} \log\left(\frac{1}{\sigma}f\left(\frac{y-\mu}{\sigma}\right)\right) \frac{1}{\sigma}f\left(\frac{y-\mu}{\sigma}\right) \mathrm{d}y$$

which, upon changing the variable back to $x = (y-\mu)/\sigma,$ produces

$$\eqalign{ H(Y) &= -\int_{-\infty}^{\infty} \log\left(\frac{1}{\sigma}f\left(x\right)\right) f\left(x\right) \mathrm{d}x \\ &= -\int_{-\infty}^{\infty} \left(\log\left(\frac{1}{\sigma}\right) + \log\left(f\left(x\right)\right)\right) f\left(x\right) \mathrm{d}x \\ &= \log\left(\sigma\right) \int_{-\infty}^{\infty} f(x) \mathrm{d}x -\int_{-\infty}^{\infty} \log\left(f\left(x\right)\right) f\left(x\right) \mathrm{d}x \\ &= \log(\sigma) + H_f. }$$

These calculations used basic properties of the logarithm, the linearity of integration, and that fact that $f(x)\mathrm{d}x$ integrates to unity (the Law of Total Probability).

The conclusion is

The entropy of $Y = X\sigma + \mu$ is the entropy of $X$ plus $\log(\sigma).$

In words, shifting a random variable does not change its entropy (we may think of the entropy as depending on the values of the probability density, but not on where those values occur), while scaling a variable (which, for $\sigma \ge 1$ "stretches" or "smears" it out) increases its entropy by $\log(\sigma).$ This supports the intuition that high-entropy distributions are "more spread out" than low-entropy distributions.


As a consequence of this result, we are free to choose convenient values of $\mu$ and $\sigma$ when computing the entropy of any distribution. For example, the entropy of a Normal$(\mu,\sigma)$ distribution can be found by setting $\mu=0$ and $\sigma=1.$ The logarithm of the density in this case is

$$\log(f(x)) = -\frac{1}{2}\log(2\pi) - x^2/2,$$

whence

$$H = -E[-\frac{1}{2}\log(2\pi) - X^2/2] = \frac{1}{2}\log(2\pi) + \frac{1}{2}.$$

Consequently the entropy of a Normal$(\mu,\sigma)$ distribution is obtained simply by adding $\log\sigma$ to this result, giving

$$H = \frac{1}{2}\log(2\pi) + \frac{1}{2} + \log(\sigma) = \frac{1}{2}\log(2\pi\,e\,\sigma^2)$$

as reported by Wikipedia.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ so there is an anyalytical link between the entropy and the moments of a distribution? stats.stackexchange.com/questions/484495/… $\endgroup$ – develarist Aug 26 at 22:07
  • 1
    $\begingroup$ @devel It's interesting that you ask that question in this context, because the present result gives a clear indication of how moments and entropy are related. In particular, because we can change the first moment of $X$ arbitrarily by adding $\mu$ without changing the entropy, there's no relationship whatever between $H$ and $\mu.$ With central moments we have the result that scaling (that is, multiplying the second central moment by $\sigma^2$) corresponds to shifting $H$ or, equivalently, scaling $\exp(H).$ Thus, relative values of $H$ give information about relative scales. $\endgroup$ – whuber Aug 27 at 11:58
  • $\begingroup$ could you give more detail on what $H_f$ is and where it came from $\endgroup$ – develarist Aug 27 at 12:04
  • $\begingroup$ @devel I don't think that's necessary, because you can determine that yourself by comparing the equation where it appears to the preceding equation. $\endgroup$ – whuber Aug 27 at 14:13
  • $\begingroup$ that's what i tried, but couldn't follow with an interpretation in terms of $H_f$ vs $H(Y)$ $\endgroup$ – develarist Aug 27 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.