How come differential entropy can be negative but mutual information can't?

Question

Differential entropy,

$$H(X) = -\int_{-\infty}^{\infty} p(x) \ln p(x) dx,$$ ordinarily is positive (the negative sign in front actually makes the entire expression positive). However, it can be negative for certain families of distributions. In other words, some densities have negative entropy. Does this only apply to differential entropy, whereas discrete entropy is the one that is always positive?

If differential entropy is included in (differential) mutual information,

$$ I(X,Y) = H(X) - H(X|Y),$$

why isn't mutual information ever negative, given that entropy might take on negative values?

or should I be asking whether there are situations where $H(X|Y) > H(X)$? Besides this, $H(X|Y) = H(X,Y) - H(Y)$, whose right-hand side would become unrealistically larger than the left-hand side if $H(Y)<0$.

Answer 1

Differential entropy is usually denoted by lowercase $h$, so I'll use that notation. Discrete entropy is always positive (because $-\log p(x)$ is always nonnegative and so all the summands in the entropy definition).

For the mutual information ($I(X;Y)=h(X)-h(X|Y)$), we always have the following inequality (Elements of Information Theory by Cover & Thomas, Second Edition, Page 256): $$h(X|Y)\leq h(X)$$ So it is nonnegative. This is intuitive because when you're given extra information, the uncertainty will either decrease or stays the same.