What's the entropy of the following generalized probability distributions?

$P_1(x) = \delta(x)$

$P_2(x,y) = \delta(x+y)$, for $0\le x\le 1$, and $P_2(x,y)=0$ otherwise.

Integrals of the type $-\int \delta(x) \ln\delta(x) \mathrm{d}x$ seem to diverge to $-\infty$ (see here). But entropy is supposed to be positive. What's going on here? How can I compute the entropy of these distributions? Is there a way to define entropy for these distributions?

  • 1
    $\begingroup$ What is your basis for assuming entropy is positive? For instance, the entropy of a Gaussian (Normal) distribution with standard deviation $\sigma$ is $\frac{1}{2}\log(2\pi e) + \log(\sigma)$ which will be extremely negative for sufficiently small $\sigma$. Are you perhaps basing your question on a different definition of entropy? $\endgroup$ – whuber Mar 27 '14 at 15:29
  • 1
    $\begingroup$ Strictly speaking, you're not actually calculating the entropy here - you're calculating the differential entropy, which is importantly different (for example, it can be negative). And no, I know of no useful definition of differential entropy which can deal with delta functions without everything going infinite. $\endgroup$ – Pat Mar 27 '14 at 16:11
  • $\begingroup$ @whuber For discrete distributions the Gibb's inequality implies that entropy is non-negative. I thought the same applied for continuous distributions? $\endgroup$ – becko Mar 27 '14 at 18:13
  • 3
    $\begingroup$ Entropy of continuous distributions behaves quite differently than that of discrete distributions, because it is defined in terms of probability densities rather than probabilities themselves. @Pat We can still make sense of (differential) entropy of delta functions; as intimated in the link in the question, it can be understood as the limiting entropy of a sequence of functions whose (compact) supports shrink to a point. Regardless of what functions are used, the entropy indeed drops to $-\infty$. This actually makes sense as "a value smaller than all real numbers." $\endgroup$ – whuber Mar 27 '14 at 18:38
  • 1
    $\begingroup$ @whuber $-\infty$ entropy can be interpreted then as infinite certainty. $\endgroup$ – becko Mar 27 '14 at 19:10

Typical Shannon entropy, on discrete set of probabilities, needs to be positive, as it is average of non-negative numbers, i.e.

$$\sum_i p_i \left(\tfrac{1}{p_i}\right).$$

Differential entropy need not to be positive. It is

$$\int p(x) \log\left(\tfrac{1}{p(x)}\right) dx,$$

which does not need to be positive. $p(x)$ is probability density, so it can be greater than one, making $\log(\tfrac{1}{p(x)})$ negative. In fact differential entropy can be viewed as Shannon entropy, where we do limit for infinitesimally small boxes and subtract $\log(1/\epsilon)$ (i.e. box size), otherwise the limit diverges:

$$ \lim_{\epsilon\to\infty} \sum_i p_{[i\epsilon, (i+1)\epsilon]} \log\left(\tfrac{1}{p_{[i\epsilon, (i+1)\epsilon]}}\right) $$ $$ \approx \lim_{\epsilon\to\infty} \sum_{i} p(i \epsilon)\epsilon \log\left(\tfrac{1}{p(i \epsilon)\epsilon}\right) $$ $$ = \lim_{\epsilon\to\infty} \left(\sum_{i} p(i \epsilon)\epsilon \log\left(\tfrac{1}{p(i \epsilon)}\right) + \log(1/\epsilon) \right) $$ $$ = \int_x p(x) \log\left(\tfrac{1}{p(x)}\right) dx + \lim_{\epsilon\to\infty}\log(1/\epsilon) $$

For Dirac delta differential entropy is $-\infty$, so you are right.

  • $\begingroup$ (+1) Doesn't $\log(\epsilon)$ need to be subtracted, as you said, rather than added? Also, because this limiting argument lacks rigor, it does not apply directly to the case of delta functions--but it still gives a nice intuition concerning why entropy and differential entropy might have different properties. $\endgroup$ – whuber May 7 '14 at 18:07
  • 1
    $\begingroup$ I meant $\log(1/\epsilon)$ (rather than $\log(\epsilon)$) needs to be subtracted (thanks for spotting that, fixed). The limiting argument is hand-waved, but can be made rigorous (for continuous functions) with the intermediate value theorem. If you know how to make it simple yet more rigorous, I would appreciate edit. $\endgroup$ – Piotr Migdal May 7 '14 at 18:20
  • 2
    $\begingroup$ Assuming Riemann integration, yes the integral can be obtained in the limit, but the separation of one limit into two also requires some justification. (In fact, it is the very failure of this operation that explains some of the problems with differential entropy.) I have no complaint about the lack of rigor for the purpose of explaining why differential entropy can be negative. However, a bit of a paradox remains: $\delta(x)$, as a discrete distribution, has entropy $0$, whereas as the limit of continuous distributions with supports shrinking to $x$, its entropy diverges to $-\infty$. $\endgroup$ – whuber May 7 '14 at 18:40
  • $\begingroup$ Isn't the correct formula for shannon/discrete entropy $H=-\sum_i p_i \log(p_i)$? It's positive because the $p_i<1 \Rightarrow \log(p_i)<0 \Rightarrow -\log(p_i)>0$. $\endgroup$ – Jake Stevens-Haas Nov 4 '18 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.