4
$\begingroup$

It's well known that if X & Y are discrete random variables X & Y (r.v.s), and $Y=g(X)$, then

$$H(X,Y)=H(X)+H(Y\mid X)=H(X),$$

where the last equality is due to $H(Y\mid X)=0.$ It also has a very intuitive interpretation - since Y is a deterministic function of X, the entropy/uncertainty of (X,Y) is the same as that of X, i.e. having Y on top of X doesn't increases uncertainty.

However, it's not very clear to me if above equalities & interpretation still hold or how to modify them, when X & Y are continuous r.v.'s? In particular, how should $h(X,Y)$ and $h(Y|X)$ be defined in this case, $-\infty$?

Thanks a lot!

$\endgroup$
5
  • $\begingroup$ Why would you expect for them to be different in continuous case? $\endgroup$
    – Tim
    Commented Sep 28, 2015 at 6:56
  • $\begingroup$ I'd like the first equality, $h(X,Y)=h(X)+h(Y\mid X)$, to hold, but I'm not certain about the 2nd one, $h(X)+h(Y\mid X)=h(X)$. $\endgroup$
    – syeh_106
    Commented Sep 28, 2015 at 7:17
  • 2
    $\begingroup$ I can't manage to make a good answer so I'll just do a comment. Conventional entropy is infinite on continuous random variables. Differential entropy solves this by computing the relative difference in entropy between the pdf and a Gaussian of variance 1 (that's one interpretation at least). Your question is poking at the fact that differential entropy on a 1D space is not the same as differential entropy on a 2D space (because the reference Gaussian is not the same !). So $H_{2D} (X,Y)$ in your case would be infinite ($-\infty$ in fact), while $H_{1D}$ is finite $H_{1D}(X,Y)=H_{1D}(X)$ $\endgroup$ Commented Sep 28, 2015 at 8:59
  • $\begingroup$ @Guillaume do you have a reference to that interpretation? $\endgroup$ Commented May 17, 2017 at 10:49
  • $\begingroup$ What I wrote is poorly explained. Discrete Random Variable entropy is hard to extend to continuous variables: if you just try to build a sequence of discrete RV that asymptotes to a continuous RV, you find that the entropy diverges. However, if you try to compute the difference in the entropies of random variables 1 and 2 in this way, the two infinities cancel out and you end up with something finite: the differential entropy. I was wrong to claim that the reference Gaussian has variance 1: the Gaussian with diffentropy 0 actually has a more complicated variance. I don't have a ref sorry. $\endgroup$ Commented May 19, 2017 at 9:26

1 Answer 1

1
$\begingroup$

It is important to notice here that differential entropy has a wildly different interpretation from the "standard" entropy of discrete variables. In particular, I would avoid putting any uncertainty-related interpretations on it.

Now, on to your questions:

Do the above equalities & interpretation still hold when X & Y are continuous r.v.'s?

As @syeh_106 points out in the comments section, the equality $h(X,Y) = h(X) + h(Y|X)$ holds perfectly fine. The equality $h(X) + h(Y|X) = h(X)$, however, doesn't. Here's a quick proof by reductio ad absurdum that $h(Y|X) \neq 0$:

We know that $I(X, Y) \geq 0$. At the same time, $I(X, Y) = h(X) - h(X|Y)$. So, if $h(Y|X)$ were $0$, we would have $h(X) \geq 0$. We know this is not the case (e.g. if X is uniformly distributed in an interval of length $L$, then $h(X) < 0$ if $L < 1$), therefore in general $h(Y|X) \neq 0$.

How should $h(X,Y)$ and $h(Y|X)$ be defined in this case, $-\infty$?

Indeed $h(X,Y) = -\infty$, although to be honest I'm not entirely sure about $h(Y|X)$ -- I think it's also $-\infty$, but it could well be the case that it's just not defined.

In any case, the fact remains that your intuition that "since $Y$ is a deterministic function of $X$, the entropy/uncertainty of $(X,Y)$ is the same as that of $X$," while correct for discrete variables, just doesn't apply to continuous variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.