11
$\begingroup$

The definition of entropy for a continuous signal is:

$$h[f] = \operatorname{E}[-\ln (f(X))] = -\int\limits_{-\infty}^{\infty} f(x) \ln (f(x))\, dx$$

According to Wikipedia, it can be negative. When would that happen? As far as I understand, $f(x)$ is always $\in[0,1]$ so $f(x)\cdot ln(f(x))$ can only be negative. What am I missing ?

$\endgroup$
2

2 Answers 2

19
$\begingroup$

Differential entropy should be considered as a measure of relative (privation of) information - not absolute. In particular, note that the differential entropy responds to a change in scale (i.e. you have a logarithm of a unitful quantity, which means that it will depend on the units you measure the axis $x$ in), which is not a concept that makes sense for a discrete information source. The inability to specify an absolute information in this context should be taken as based on the intuitive idea that the amount of information required to specify a specific value in an infinite continuum is itself infinite as you must distinguish one from amongst an infinitude of possibilities to achieve such a specification.

To understand it more precisely, consider a uniform distribution where the differential entropy is zero. A simple example is precisely that which is one unit wide:

$$P(x) = \begin{cases}1,\ \mbox{if $x \in [0, 1]$} \\ 0,\ \mbox{otherwise} \end{cases}$$

If you compute the differential entropy, $h$, of the above distribution, you will find it is zero since $\ln(1) = 0$ and moreover in the appropriate limit $0 \ln(0)$ "equals" 0. This corresponds to the fact you (or the agent to which the Bayesian probability $P(x)$ is specified relative to) know the position (or whatever) of the object to within exactly one unit. If you make the distribution wider, say two units, so you have even less information, then the differential entropy will be $\ln(2)$ or about 0.693. This is the same as the discrete entropy for an infinite discrete set of bins, each standing in for "a unit", or if you like, the bins between marks on a ruler and where you only report the measurement made with the ruler as a whole number of its finest ticks, now distributed uniformly among two such bins, and means we have 0.697 nats less information now about the position of the particle up to a resolution of one unit.

Negative differential entropy then just means we go the other way - since we aren't working with discrete bins, we can know it "more precisely" than one bin, i.e. to an accuracy less than one unit, and thus the entropy (privation of information) will have to be less now as we are more informed, thus now less than zero. But if I switch to a finer scale, i.e. a smaller unit, then the entropy will once again exceed zero, as now we don't have enough to know it down to that fine scale.

You cannot have an absolute measure because on a continuum, effectively you have an uncountable infinity of "bins" within any arbitrarily small interval, thus even a tiny interval of uncertainty is still effectively infinite information. Thus we have to get up "high into infinity" to measure the differences in entropy in realistic distributions and that is why the "bottom" of differential entropy is at $-\infty$, which is like a probability zero in a continuous probability measure, and how that such does not necessarily represent impossibility, but rather negligibility with regard to the infinitude of the sets we are considering.

Or to go the other way intuitively, suppose you were treating the continuum as a set of bins - one for each point - like how you do an ordinary discrete random variable. Then with a probability distribution of $P(x) = \delta(x - a)$, i.e. a delta function at some central real number $a$, that is one bin occupied, so entropy 0, but if you now have 2 bins with probability 1/2, i.e. $P(x) = \frac{1}{2} \delta(x - a_1) + \frac{1}{2} \delta(x - a_2)$ meaning "we know the particle is at either $a_1$ exactly or $a_2$ exactly but not which", then by the usual discrete entropy formula you have entropy $0.693$ nat (or $1$ shannon using $\lg$ instead of $\ln$). But if you continue in this way, long before you reach a truly continuous distribution you will soon "top out" (after a countably infinite number of bins have been summed) using discrete entropy, saturating at positive infinity. And that is what I mean by saying that to then go up into continuous distributions, you then have to "soar up high" - effectively the integral jams in an uncountable infinity and rises your reference point far above the true baseline so you can distinguish the different infinite amounts that are "above the infinite boundary of the discrete entropy". That rise then, by symmetry, puts the baseline infinitely far below you, or at $-\infty$, and moreover due to the Archimedean nature of the real numbers, prevents you from distinguishing these finite-bin cases any more (all of them, if you check for yourself, have differential entropy $-\infty$).

$\endgroup$
7
$\begingroup$

You are just confusing $f(x)$ and $F(X)$. The density function $f(x)$ can be greater 1. It just integrates to 1. It is $F(X) \in [0, 1]$.

Best

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.