# Continuous joint entropy with fully dependent variable

Consider a variable $X$ with a continuous uniform distribution in the interval $[a,b]$ and a variable $Y$ that is fully dependent on $X$, i.e., $p(Y=y\ |\ X=x) = \delta (x-y)$, where $\delta$ is a delta distribution with peak $x$. What is the entropy $H(X,Y)$ of the joint distribution?

Intuitively, samples from the joint distribution should have a uniform distribution along the diagonal in $[a,b]^2$, so the entropy should be finite, but I can't really figure it out.

• I think it should be $-\infty$, because the distribution lies in a 1-dim. subspace, but I would really appreciate a formal proof. – ASML Apr 11 '15 at 3:27
• What do you mean by a formal proof? (It is straightforward calculation.) – Piotr Migdal Apr 18 '15 at 13:10
• If it is straightforward, I'd appreciate an answer. If the answer is indeed -$\infty$, then I still think it's counterintuitive, because visually there's clearly some finite uncertainty, because the data samples are spread along the diagonal. – ASML Apr 19 '15 at 15:08

First, let's get the probability distribution:

$$p(x,y) = p(y|x) p(x) = \delta(y - x) p(x)$$

Then, remembering that Dirac delta works as follows: $$\int \delta(y - x) f(y)dy= f(x)$$ we perform the calculation: $$H(X,Y) = - \int p(x,y) [\log p(x,y)] dxdy\\ = - \int \delta(y - x) p(x) \log[\delta(y - x) p(x)] dx dy\\ = - \int p(x) \log[\delta(x - x) p(x)] dx\\ = -\infty - \int p(x) \log[ p(x)] dx\\ = -\infty.$$

And, in general, if probability distribution occupies manifold of less dimensions than space we use, it's differential entropy is $-\infty$.

Why it should not surprise you?

One way of looking at it is the following - entropy is related to the uncertainty of space occupied by the probability distribution. It is normalized in such way than unit (hyper)cube with uniform probability distribution gives entropy $0$. If its volume is infinitely smaller, its entropy its infinitely lower.

Other way is too look at the (differential) mutual information, i.e.: $$I(X;Y) = H(X) + H(Y) - H(X, Y).$$ Since there are continuous variables, we can transmit information of infinite length just with one trial! (E.g. as digits of a real number.) That way, even though $H(X)$ and $X(Y)$ are finite, $H(X, Y)$ needs to be minus infinity, so that $I(X;Y)=\infty$.

• Thank you, that helped me a lot! I understand your calculation and I'm convinced that it's correct. Still: An entropy of minus infinity sounds as if it was a deterministic process (no uncertainty), which it is not. I find it surprising that H(X) is finite and H(X,Y) is not. The uncertainty of X should carry over to the joint entropy somehow...only from an intuitive point of view, mathematically it's clear now. – ASML Apr 20 '15 at 3:24
• @ASML Differential entropy has some other properties than discrete entropy (where 0 is iff there is no randomness); and adding any amount of deterministic outcome makes it $\-\infty$ (e.g. entropy of $\tfrac{1}{2}+\tfrac{1}{2}\delta(x-\tfrac{1}{2})$ on unit interval). – Piotr Migdal Apr 20 '15 at 9:07