Continuous joint entropy with fully dependent variable Consider a variable $X$ with a continuous uniform distribution in the interval $[a,b]$ and a variable $Y$ that is fully dependent on $X$, i.e., $p(Y=y\ |\ X=x) = \delta (x-y)$, where $\delta$ is a delta distribution with peak $x$. What is the entropy $H(X,Y)$ of the joint distribution?
Intuitively, samples from the joint distribution should have a uniform distribution along the diagonal in $[a,b]^2$, so the entropy should be finite, but I can't really figure it out.
 A: First, let's get the probability distribution:
$$p(x,y) = p(y|x) p(x) = \delta(y - x) p(x)$$
Then, remembering that Dirac delta works as follows:
$$\int \delta(y - x) f(y)dy= f(x)$$
we perform the calculation:
$$
H(X,Y) = - \int p(x,y) [\log p(x,y)] dxdy\\
= - \int \delta(y - x) p(x) \log[\delta(y - x) p(x)] dx dy\\
= - \int p(x) \log[\delta(x - x) p(x)] dx\\
= -\infty - \int p(x) \log[ p(x)] dx\\
= -\infty.
$$
And, in general, if probability distribution occupies manifold of less dimensions than space we use, it's differential entropy is $-\infty$.
Why it should not surprise you?
One way of looking at it is the following - entropy is related to the uncertainty of space occupied by the probability distribution. It is normalized in such way than unit (hyper)cube with uniform probability distribution gives entropy $0$. If its volume is infinitely smaller, its entropy its infinitely lower.
Other way is too look at the (differential) mutual information, i.e.:
$$I(X;Y) = H(X) + H(Y) - H(X, Y).$$
Since there are continuous variables, we can transmit information of infinite length just with one trial! (E.g. as digits of a real number.)
That way, even though $H(X)$ and $X(Y)$ are finite, $H(X, Y)$ needs to be minus infinity, so that $I(X;Y)=\infty$.
