# joint differential entropy $h(X,Y)$, when $Y=g(X)$

It's well known that if X & Y are discrete random variables X & Y (r.v.s), and $Y=g(X)$, then

$$H(X,Y)=H(X)+H(Y\mid X)=H(X),$$

where the last equality is due to $H(Y\mid X)=0.$ It also has a very intuitive interpretation - since Y is a deterministic function of X, the entropy/uncertainty of (X,Y) is the same as that of X, i.e. having Y on top of X doesn't increases uncertainty.

However, it's not very clear to me if above equalities & interpretation still hold or how to modify them, when X & Y are continuous r.v.'s? In particular, how should $h(X,Y)$ and $h(Y|X)$ be defined in this case, $-\infty$?

Thanks a lot!

• Why would you expect for them to be different in continuous case? – Tim Sep 28 '15 at 6:56
• I'd like the first equality, $h(X,Y)=h(X)+h(Y\mid X)$, to hold, but I'm not certain about the 2nd one, $h(X)+h(Y\mid X)=h(X)$. – syeh_106 Sep 28 '15 at 7:17
• I can't manage to make a good answer so I'll just do a comment. Conventional entropy is infinite on continuous random variables. Differential entropy solves this by computing the relative difference in entropy between the pdf and a Gaussian of variance 1 (that's one interpretation at least). Your question is poking at the fact that differential entropy on a 1D space is not the same as differential entropy on a 2D space (because the reference Gaussian is not the same !). So $H_{2D} (X,Y)$ in your case would be infinite ($-\infty$ in fact), while $H_{1D}$ is finite $H_{1D}(X,Y)=H_{1D}(X)$ – Guillaume Dehaene Sep 28 '15 at 8:59
• @Guillaume do you have a reference to that interpretation? – kjetil b halvorsen May 17 '17 at 10:49
• What I wrote is poorly explained. Discrete Random Variable entropy is hard to extend to continuous variables: if you just try to build a sequence of discrete RV that asymptotes to a continuous RV, you find that the entropy diverges. However, if you try to compute the difference in the entropies of random variables 1 and 2 in this way, the two infinities cancel out and you end up with something finite: the differential entropy. I was wrong to claim that the reference Gaussian has variance 1: the Gaussian with diffentropy 0 actually has a more complicated variance. I don't have a ref sorry. – Guillaume Dehaene May 19 '17 at 9:26

It is important to notice here that differential entropy has a wildly different interpretation from the "standard" entropy of discrete variables. In particular, I would avoid putting any uncertainty-related interpretations on it.

As @syeh_106 points out in the comments section, the equality $h(X,Y) = h(X) + h(Y|X)$ holds perfectly fine. The equality $h(X) + h(Y|X) = h(X)$, however, doesn't. Here's a quick proof by reductio ad absurdum that $h(Y|X) \neq 0$:
We know that $I(X, Y) \geq 0$. At the same time, $I(X, Y) = h(X) - h(X|Y)$. So, if $h(Y|X)$ were $0$, we would have $h(X) \geq 0$. We know this is not the case (e.g. if X is uniformly distributed in an interval of length $L$, then $h(X) < 0$ if $L < 1$), therefore in general $h(Y|X) \neq 0$.
How should $h(X,Y)$ and $h(Y|X)$ be defined in this case, $-\infty$?
Indeed $h(X,Y) = -\infty$, although to be honest I'm not entirely sure about $h(Y|X)$ -- I think it's also $-\infty$, but it could well be the case that it's just not defined.
In any case, the fact remains that your intuition that "since $Y$ is a deterministic function of $X$, the entropy/uncertainty of $(X,Y)$ is the same as that of $X$," while correct for discrete variables, just doesn't apply to continuous variables.