How to interpret differential entropy?

Question

I recently read this article on the entropy of a discrete probability distribution. It describes a nice way of thinking about entropy as the expected number bits (at least when using $\log_2$ in your entropy definition) needed to encode a message when your encoding is optimal, given the probability distribution of the words you use.

However, when extending to the continuous case like here I believe this way of thinking breaks down, since $\sum_x p(x) = \infty$ for any continuous probability distribution $p(x)$ (please correct me if that is wrong), so I was wondering if there is nice way of thinking about what continuous entropy means, just like with the discrete case.

Answer 1

There is no interpretation of differential entropy which would be as meaningful or useful as that of entropy. The problem with continuous random variables is that their values typically have 0 probability, and therefore would require an infinite number of bits to encode.

If you look at the limit of discrete entropy by measuring the probability of intervals $[n\varepsilon, (n + 1)\varepsilon[$, you end up with

$$-\int p(x) \log_2 p(x) \, dx - \log_2 \varepsilon$$

and not the differential entropy. This quantity is in a sense more meaningful, but will diverge to infinity as we take smaller and smaller intervals. It makes sense, since we'll need more and more bits to encode in which of the many intervals the value of our random value falls.

A more useful quantity to look at for continuous distributions is the relative entropy (also Kullback-Leibler divergence). For discrete distributions:

$$D_\text{KL}[P || Q] = \sum_x P(x) \log_2 \frac{P(x)}{Q(x)}.$$

It measures the number of extra bits used when the true distribution is $P$, but we use $-\log Q_2(x)$ bits to encode $x$. We can take the limit of relative entropy and arrive at

$$D_\text{KL}[p \mid\mid q] = \int p(x) \log_2 \frac{p(x)}{q(x)} \, dx,$$

because $\log_2 \varepsilon$ will cancel. For continuous distributions this corresponds to the number of extra bits used in the limit of infinitesimally small bins. For both continuous and discrete distributions, this is always non-negative.

Now, we could think of differential entropy as the negative relative entropy between $p(x)$ and an unnormalized density $\lambda(x) = 1$,

$$-\int p(x) \log_2 p(x) \, dx = -D_\text{KL}[p \mid\mid \lambda].$$

Its interpretation would be the difference in the number of bits required by using $-\log_2 \int_{n\varepsilon}^{(n + 1)\varepsilon} p(x) \, dx$ bits to encode the $n$-th interval instead of $-\log \varepsilon$ bits. Even though the former would be optimal, this difference can now be negative, because $\lambda$ is cheating (by not integrating to 1) and therefore might assign fewer bits on average than theoretically possible.

See Sergio Verdu's talk for a great introduction to relative entropy.

Answer 2

For the differential entropy there also exists another, more mathematical interpretation, which is closely related to the bit-interpretation for the entropy.

The differential entropy describes the equivalent side length (in logs) of the set that contains most of the probability of the distribution.

This is nicely illustrated and explained in Theorem 8.2.3 in Elements of Information Theory by Thomas M. Cover, Joy A. Thomas

Intuitive Explanation

In non-rigorous terms, this statement means the following: Let's assume we have a multivariate probability distribution with entropy $h$. The side length of a volume that entails most of the probability mass of this distribution (apart from a negligible amount), can be described by some volume.

If we assume we describe this volume by some hypercube with sides of equal length (= equivalent side lengths), then this side length is equal to $2^h$.

Intuitively this means, that if we have a low entropy, the probability mass of the distribution is confined to a small area. Vice versa, high entropy tells us that the probability mass is spread widely across a large area.

Mathematical View

In actual notation, the theorem states the following

$1 - \epsilon 2^{n(h(X) - \epsilon)} \leq \text{Vol}(A_{\epsilon}^{(n)}) \leq 2^{n(h(X) + \epsilon)}$, where $X$ is a random variable with the distribution of interest, $\epsilon$ is a real number, $A_{\epsilon}^{(n)}$ is a set, $h(X)$ is the differential entropy of $X$ and $n$ (required to be large) is the dimension of $X$.

This implies that "$A_{\epsilon}^{(n)}$ is the smallest volume set with probability $1-\epsilon$, to first order in the exponent." (Elements of Information Theory by Thomas M. Cover, Joy A. Thomas, Wiley, Second Edition, 2006)

Relation to entropy of discrete probability distributions

This interpretation of differential entropy is closely related to the entropy for discrete distributions.

Discrete Case: As OP stated, the entropy tells us how many bits are needed to encode a message given a probability distribution over words.

Continuous case: Here we are dealing with continuous support. For example, let's assume the support is on the real line $\mathbb{R}$. The differential entropy tells us, how long the interval on the real line has to be to capture almost all information contained in the probability distribution.

• If we have a widely spread distribution -> the entropy will be high
• If we have a sharp distribution, most probability mass will be in a small interval -> the entropy will be low.

Example with $N(0,1)$

The entropy of a standard normal distribution with $\sigma^2 = 1$ is

$\frac{1}{2}\text{ln}(2\pi \sigma^2) + \frac{1}{2} = \frac{1}{2}\text{ln}( 2 \pi) + \frac{1}{2}$

We can visualize this with a small code example in Python:

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

ys = np.random.normal(size = 10000)
h = 0.5*np.log(2*np.pi*np.exp(1)) 
side_length = 2**h

sns.kdeplot(ys, fill = True)
plt.vlines(x = side_length/2, ymin = 0, ymax = 0.4, color = 'red', linestyles = 'dashed')
plt.vlines(x = -side_length/2, ymin = 0, ymax = 0.4, color = 'red', linestyles = 'dashed')

This side length captures a large portion of the probability mass in this distribution:

The interval between the red lines is $2^h$. As in this case $n$ is only 1 (and the Theorem above requires $n$ to be large), we can clearly see that the entropy is not exactly the equivalent side length of the volume that captures almost all probability mass.

This graph also explains why for the Gaussian, the mean does not affect the differential entropy: No matter where I shift the distribution to - the equivalent side length will stay the same and is only influenced by the variance.