3
$\begingroup$

It's well known that if X & Y are discrete random variables X & Y (r.v.s), and $Y=g(X)$, then

$$H(X,Y)=H(X)+H(Y\mid X)=H(X),$$

where the last equality is due to $H(Y\mid X)=0.$ It also has a very intuitive interpretation - since Y is a deterministic function of X, the entropy/uncertainty of (X,Y) is the same as that of X, i.e. having Y on top of X doesn't increases uncertainty.

However, it's not very clear to me if above equalities & interpretation still hold or how to modify them, when X & Y are continuous r.v.'s? In particular, how should $h(X,Y)$ and $h(Y|X)$ be defined in this case, $-\infty$?

Thanks a lot!

$\endgroup$
  • $\begingroup$ Why would you expect for them to be different in continuous case? $\endgroup$ – Tim Sep 28 '15 at 6:56
  • $\begingroup$ I'd like the first equality, $h(X,Y)=h(X)+h(Y\mid X)$, to hold, but I'm not certain about the 2nd one, $h(X)+h(Y\mid X)=h(X)$. $\endgroup$ – syeh_106 Sep 28 '15 at 7:17
  • 2
    $\begingroup$ I can't manage to make a good answer so I'll just do a comment. Conventional entropy is infinite on continuous random variables. Differential entropy solves this by computing the relative difference in entropy between the pdf and a Gaussian of variance 1 (that's one interpretation at least). Your question is poking at the fact that differential entropy on a 1D space is not the same as differential entropy on a 2D space (because the reference Gaussian is not the same !). So $H_{2D} (X,Y)$ in your case would be infinite ($-\infty$ in fact), while $H_{1D}$ is finite $H_{1D}(X,Y)=H_{1D}(X)$ $\endgroup$ – Guillaume Dehaene Sep 28 '15 at 8:59
  • $\begingroup$ @Guillaume do you have a reference to that interpretation? $\endgroup$ – kjetil b halvorsen May 17 '17 at 10:49
  • $\begingroup$ What I wrote is poorly explained. Discrete Random Variable entropy is hard to extend to continuous variables: if you just try to build a sequence of discrete RV that asymptotes to a continuous RV, you find that the entropy diverges. However, if you try to compute the difference in the entropies of random variables 1 and 2 in this way, the two infinities cancel out and you end up with something finite: the differential entropy. I was wrong to claim that the reference Gaussian has variance 1: the Gaussian with diffentropy 0 actually has a more complicated variance. I don't have a ref sorry. $\endgroup$ – Guillaume Dehaene May 19 '17 at 9:26
1
$\begingroup$

It is important to notice here that differential entropy has a wildly different interpretation from the "standard" entropy of discrete variables. In particular, I would avoid putting any uncertainty-related interpretations on it.

Now, on to your questions:

Do the above equalities & interpretation still hold when X & Y are continuous r.v.'s?

As @syeh_106 points out in the comments section, the equality $h(X,Y) = h(X) + h(Y|X)$ holds perfectly fine. The equality $h(X) + h(Y|X) = h(X)$, however, doesn't. Here's a quick proof by reductio ad absurdum that $h(Y|X) \neq 0$:

We know that $I(X, Y) \geq 0$. At the same time, $I(X, Y) = h(X) - h(X|Y)$. So, if $h(Y|X)$ were $0$, we would have $h(X) \geq 0$. We know this is not the case (e.g. if X is uniformly distributed in an interval of length $L$, then $h(X) < 0$ if $L < 1$), therefore in general $h(Y|X) \neq 0$.

How should $h(X,Y)$ and $h(Y|X)$ be defined in this case, $-\infty$?

Indeed $h(X,Y) = -\infty$, although to be honest I'm not entirely sure about $h(Y|X)$ -- I think it's also $-\infty$, but it could well be the case that it's just not defined.

In any case, the fact remains that your intuition that "since $Y$ is a deterministic function of $X$, the entropy/uncertainty of $(X,Y)$ is the same as that of $X$," while correct for discrete variables, just doesn't apply to continuous variables.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.