2
$\begingroup$

Differential entropy,

$$H(X) = -\int_{-\infty}^{\infty} p(x) \ln p(x) dx,$$ ordinarily is positive (the negative sign in front actually makes the entire expression positive). However, it can be negative for certain families of distributions. In other words, some densities have negative entropy. Does this only apply to differential entropy, whereas discrete entropy is the one that is always positive?

If differential entropy is included in (differential) mutual information,

$$ I(X,Y) = H(X) - H(X|Y),$$

why isn't mutual information ever negative, given that entropy might take on negative values?

or should I be asking whether there are situations where $H(X|Y) > H(X)$? Besides this, $H(X|Y) = H(X,Y) - H(Y)$, whose right-hand side would become unrealistically larger than the left-hand side if $H(Y)<0$.

$\endgroup$
2
  • $\begingroup$ Can you give an example of a distribution where differential entropy is negative? $\endgroup$
    – Alexis
    Sep 8, 2020 at 4:38
  • 2
    $\begingroup$ Uniform distribution with width < 1 can be an example $\endgroup$
    – gunes
    Sep 8, 2020 at 10:36

1 Answer 1

3
$\begingroup$

Differential entropy is usually denoted by lowercase $h$, so I'll use that notation. Discrete entropy is always positive (because $-\log p(x)$ is always nonnegative and so all the summands in the entropy definition).

For the mutual information ($I(X;Y)=h(X)-h(X|Y)$), we always have the following inequality (Elements of Information Theory by Cover & Thomas, Second Edition, Page 256): $$h(X|Y)\leq h(X)$$ So it is nonnegative. This is intuitive because when you're given extra information, the uncertainty will either decrease or stays the same.

$\endgroup$
9
  • $\begingroup$ Could you show more explicitly how this answers the question(s)? $\endgroup$ Sep 8, 2020 at 9:36
  • $\begingroup$ I've added a reference for the inequality @RichardHardy $\endgroup$
    – gunes
    Sep 8, 2020 at 10:41
  • 2
    $\begingroup$ Thank you. I am not challenging your claims. I only meant to ask what I can conclude from what you wrote. E.g. what is the answer to the question How come differential entropy can be negative but mutual information can't? and other ones in the post? $\endgroup$ Sep 8, 2020 at 10:53
  • $\begingroup$ Yeah, the answer actually relied on the equation in the OP $\endgroup$
    – gunes
    Sep 8, 2020 at 11:01
  • 2
    $\begingroup$ Let the OP have a say on that. I just tried to make the answer as clear as possible. Sometimes what is obvious for the answerer is not obvious for someone else, so I prefer very explicit answers. Thank you for your help! $\endgroup$ Sep 8, 2020 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.