For a pdf $f(x)$ (i.e. continuous distribution), Entropy (differential entropy) is defined as:
$H_C(X) = -\int_\mathbb{X} f(x)\log f(x)\,dx.$
For a discrete distribution with p.m.f $F(x)$, Entropy is defined as:
$H_D(X) = -\sum_{i=1}^n {F(x_i) \log F(x_i)}.$
The definitions look analogous to each other. However, entropy increases with dispersion for continuous but not for discrete distributions. Why?
It seems you might be asking why spreading discrete data has no effect on entropy. Because entropy is a measure of expected surprise, the various labels or values that a thing can take is immaterial. So, the discrete values $x_i$ don't matter, merely their masses and spreading the $x_i$s has no effect.
In the continuous case, spreading things out by scaling inevitably reduces the densities, which affects the entropy as defined in your question. The definition is consistent with our intuition of entropy, Shannon explains, because we typically compare two entropies, and since both are scaled, this effect cancels out. Differential entropy is also consistent with discrete entropy in the sense that it approximates what would happen if the entropy of the quantized distribution were measured.
Note that in the continuous case, spreading things out by other methods can leave entropy unchanged. For example, a uniform distribution over $[-\frac12,\frac12]$ has entropy zero. "Spreading it out" so that it is uniform over $[-10, -9.5], [9.5,10]$ still has entropy zero. Spreading never matters; only the expected surprisal does.
There is one important difference between the continuous and discrete entropies. In the discrete case the entropy measures in an absolute way the randomness of the chance variable. In the continuous case the measurement is relative to the coordinate system [and] the entropy can be considered a measure of randomness relative to an assumed standard, namely the coordinate system chosen with each small volume element $dx_1, …, dx_n$ given equal weight. When we change the coordinate system to $y_1, …, y_n$, the entropy in the new system measures the randomness when equal volume elements $dx_1, …, dx_n$ in the new system are given equal weight.
In spite of this dependence on the coordinate system the entropy concept is as important in the continuous case as the discrete case. This is due to the fact that the derived concepts of information rate and channel capacity depend on the difference of two entropies and this difference does not depend on the coordinate frame, each of the two terms being changed by the same amount.
The entropy of a continuous distribution can be negative. The scale of measurements sets an arbitrary zero corresponding to a uniform distribution over a unit volume. A distribution which is more confined than this has less entropy and will be negative. The rates and capacities will, however, always be nonnegative. — Shannon 1948
In the continuous case, because they are continuous, spreading out the $x$ requires that you dampen the densities, and so this effects the entropy, since the density is tied in an explicit way to the values of $x$.
In the discrete case, the values of $x_i$ are more like indices, and there is no explicit connection between the density and the $x_i$ (unlike for continuous, where there is a function connecting them), so spreading out the $x_i$ doesn't affect the densities, and hence doesn't affect the entropy.
All that being said, it was my understanding that the entropy of a continuous distribution (with unbounded support) tends to diverges...
