Consider a variable $X$ with a continuous uniform distribution in the interval $[a,b]$ and a variable $Y$ that is fully dependent on $X$, i.e., $p(Y=y\ |\ X=x) = \delta (x-y)$, where $\delta$ is a delta distribution with peak $x$. What is the entropy $H(X,Y)$ of the joint distribution?

Intuitively, samples from the joint distribution should have a uniform distribution along the diagonal in $[a,b]^2$, so the entropy should be finite, but I can't really figure it out.

  • $\begingroup$ I think it should be $-\infty$, because the distribution lies in a 1-dim. subspace, but I would really appreciate a formal proof. $\endgroup$ – ASML Apr 11 '15 at 3:27
  • $\begingroup$ What do you mean by a formal proof? (It is straightforward calculation.) $\endgroup$ – Piotr Migdal Apr 18 '15 at 13:10
  • $\begingroup$ If it is straightforward, I'd appreciate an answer. If the answer is indeed -$\infty$, then I still think it's counterintuitive, because visually there's clearly some finite uncertainty, because the data samples are spread along the diagonal. $\endgroup$ – ASML Apr 19 '15 at 15:08

First, let's get the probability distribution:

$$p(x,y) = p(y|x) p(x) = \delta(y - x) p(x)$$

Then, remembering that Dirac delta works as follows: $$\int \delta(y - x) f(y)dy= f(x)$$ we perform the calculation: $$ H(X,Y) = - \int p(x,y) [\log p(x,y)] dxdy\\ = - \int \delta(y - x) p(x) \log[\delta(y - x) p(x)] dx dy\\ = - \int p(x) \log[\delta(x - x) p(x)] dx\\ = -\infty - \int p(x) \log[ p(x)] dx\\ = -\infty. $$

And, in general, if probability distribution occupies manifold of less dimensions than space we use, it's differential entropy is $-\infty$.

Why it should not surprise you?

One way of looking at it is the following - entropy is related to the uncertainty of space occupied by the probability distribution. It is normalized in such way than unit (hyper)cube with uniform probability distribution gives entropy $0$. If its volume is infinitely smaller, its entropy its infinitely lower.

Other way is too look at the (differential) mutual information, i.e.: $$I(X;Y) = H(X) + H(Y) - H(X, Y).$$ Since there are continuous variables, we can transmit information of infinite length just with one trial! (E.g. as digits of a real number.) That way, even though $H(X)$ and $X(Y)$ are finite, $H(X, Y)$ needs to be minus infinity, so that $I(X;Y)=\infty$.

| cite | improve this answer | |
  • $\begingroup$ Thank you, that helped me a lot! I understand your calculation and I'm convinced that it's correct. Still: An entropy of minus infinity sounds as if it was a deterministic process (no uncertainty), which it is not. I find it surprising that H(X) is finite and H(X,Y) is not. The uncertainty of X should carry over to the joint entropy somehow...only from an intuitive point of view, mathematically it's clear now. $\endgroup$ – ASML Apr 20 '15 at 3:24
  • 1
    $\begingroup$ @ASML Differential entropy has some other properties than discrete entropy (where 0 is iff there is no randomness); and adding any amount of deterministic outcome makes it $\-\infty$ (e.g. entropy of $\tfrac{1}{2}+\tfrac{1}{2}\delta(x-\tfrac{1}{2})$ on unit interval). $\endgroup$ – Piotr Migdal Apr 20 '15 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.