3
$\begingroup$

For a pdf $f(x)$ (i.e. continuous distribution), Entropy (differential entropy) is defined as:

$H_C(X) = -\int_\mathbb{X} f(x)\log f(x)\,dx.$

For a discrete distribution with p.m.f $F(x)$, Entropy is defined as:

$H_D(X) = -\sum_{i=1}^n {F(x_i) \log F(x_i)}.$

The definitions look analogous to each other. However, entropy increases with dispersion for continuous but not for discrete distributions. Why?

$\endgroup$
  • $\begingroup$ Please make more concrete the expression "increases with dispersion": increases as variance increases? Some other measure of dispersion? Increases as the support gets larger? $\endgroup$ – Alecos Papadopoulos Oct 27 '13 at 21:25
  • $\begingroup$ Thanks @AlecosPapadopoulos - We can focus on variance, but I think the same happens with IQR and other measures of dispersion. $\endgroup$ – Josh Oct 27 '13 at 21:36
2
$\begingroup$

It seems you might be asking why spreading discrete data has no effect on entropy. Because entropy is a measure of expected surprise, the various labels or values that a thing can take is immaterial. So, the discrete values $x_i$ don't matter, merely their masses and spreading the $x_i$s has no effect.

In the continuous case, spreading things out by scaling inevitably reduces the densities, which affects the entropy as defined in your question. The definition is consistent with our intuition of entropy, Shannon explains, because we typically compare two entropies, and since both are scaled, this effect cancels out. Differential entropy is also consistent with discrete entropy in the sense that it approximates what would happen if the entropy of the quantized distribution were measured.

Note that in the continuous case, spreading things out by other methods can leave entropy unchanged. For example, a uniform distribution over $[-\frac12,\frac12]$ has entropy zero. "Spreading it out" so that it is uniform over $[-10, -9.5], [9.5,10]$ still has entropy zero. Spreading never matters; only the expected surprisal does.

There is one important difference between the continuous and discrete entropies. In the discrete case the entropy measures in an absolute way the randomness of the chance variable. In the continuous case the measurement is relative to the coordinate system [and] the entropy can be considered a measure of randomness relative to an assumed standard, namely the coordinate system chosen with each small volume element $dx_1, …, dx_n$ given equal weight. When we change the coordinate system to $y_1, …, y_n$, the entropy in the new system measures the randomness when equal volume elements $dx_1, …, dx_n$ in the new system are given equal weight.

In spite of this dependence on the coordinate system the entropy concept is as important in the continuous case as the discrete case. This is due to the fact that the derived concepts of information rate and channel capacity depend on the difference of two entropies and this difference does not depend on the coordinate frame, each of the two terms being changed by the same amount.

The entropy of a continuous distribution can be negative. The scale of measurements sets an arbitrary zero corresponding to a uniform distribution over a unit volume. A distribution which is more confined than this has less entropy and will be negative. The rates and capacities will, however, always be nonnegative. — Shannon 1948

$\endgroup$
  • 1
    $\begingroup$ Thanks although I still fail to see how dispersion increases for one and not for the other, based on their mathematical definition. What part of the text you quoted explains this and how? $\endgroup$ – Josh Oct 27 '13 at 22:11
  • $\begingroup$ @Josh: Are you asking why spreading the values that discrete distribution takes on has no effect on entropy? $\endgroup$ – Neil G Oct 27 '13 at 22:14
  • $\begingroup$ Thanks Neil - Yes, why spreading the distribution increases entropy in the continuous setting but not in the discrete case. $\endgroup$ – Josh Oct 27 '13 at 22:20
  • $\begingroup$ @Josh: I'm still not sure what you're asking. I've updated my answer to answer a couple possible "whys". $\endgroup$ – Neil G Oct 27 '13 at 22:43
  • 1
    $\begingroup$ Thanks Neil - But couldn't I say the same about the continuous case? The $x$ don't show up in the $H_C(X)$ equation either, except to find their densities with $f(x)$ $\endgroup$ – Josh Oct 27 '13 at 23:24
1
$\begingroup$

In the continuous case, because they are continuous, spreading out the $x$ requires that you dampen the densities, and so this effects the entropy, since the density is tied in an explicit way to the values of $x$.

In the discrete case, the values of $x_i$ are more like indices, and there is no explicit connection between the density and the $x_i$ (unlike for continuous, where there is a function connecting them), so spreading out the $x_i$ doesn't affect the densities, and hence doesn't affect the entropy.

All that being said, it was my understanding that the entropy of a continuous distribution (with unbounded support) tends to diverges...

$\endgroup$
  • $\begingroup$ "In the continuous case, because they are continuous, spreading out the x requires that you dampen the densities" — not necessarily… $\endgroup$ – Neil G Oct 28 '13 at 3:02
  • $\begingroup$ how not? we must preserve continuity... $\endgroup$ – Ethan Oct 28 '13 at 15:48
  • $\begingroup$ The asker specifies in a comment that by spreading out he means increasing the variance. Consider a uniform distribution on [-10,-9] and [9,10]. You could spread it out so that it is uniform on [-20,-19] and [19,20]. The variance is larger, but the entropy is unchanged. Also the densities are not dampened. $\endgroup$ – Neil G Oct 28 '13 at 19:32
  • $\begingroup$ touche. will that work if we require the support be fully continuous (not just piecewise)? $\endgroup$ – Ethan Oct 30 '13 at 0:59
  • 1
    $\begingroup$ yes, you could have a Gaussian mixture model having two modes whereby you "spread it out" by moving the modes away from each other while reducing the variance of each mode. The overall variance goes up, but the entropy goes down. $\endgroup$ – Neil G Oct 30 '13 at 1:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.