A textbook I am reading states that$$H(X,Y)=H(X)+H(Y|X)$$where $H(X,Y)$ is the joint entropy of random variables $X,Y$, $H(X)$ the entropy of $X$, and $H(Y|X)$ is conditional entropy. It then states that in a similar way, one can show $$H(X,Y|Z)=H(X|Z)+H(Y|X,Z).$$ I am confused how to interpret the term on the left hand-side. Is it meant to be interpreted as the cross entropy of the distribution of $X$ and the conditional distribution of $Y$ conditioned on $X$, or is it meant to be interpreted as the conditional entropy of the joint distribution of $X$ and $Y$, conditioned on $Z$?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ Adding the definition oft $H$ would probably be helpful. $\endgroup$– picky_porpoiseCommented Apr 20 at 8:01
-
$\begingroup$ For a random variable $X$, with distribution $p$, $H(X)=\sum\limits_xp(x)ln(\frac{1}{p(x)})$ $\endgroup$– user124910Commented Apr 20 at 16:21
-
$\begingroup$ Maybe move that into the question. Also what about $H(X,Y)$? $\endgroup$– picky_porpoiseCommented Apr 20 at 16:23
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It is the conditional entropy involving three random variables, namely
$$\mathrm H(X, Y\mid Z):=-\sum_{i, j, k}p(x_i, y_j, z_k) \log p(x_i, y_j\mid z_k); $$
This is the "uncertainty" about $X$ and $Y$ given $Z.$
(The other being $$\mathrm H( Y\mid X, Z):=-\sum_{i, j, k}p(x_i, y_j, z_k) \log p( y_j\mid x_i, z_k)$$ which measures the "uncertainty" about $Y$ given $X$ and $Z.$)
The proof of the result is based on writing $p(Y\mid X, Z) $ in terms of $p(X, Y\mid Z) $ and $p(X\mid Z), $ which is nothing but the application of definition of conditional probability. See this Math.SE post.
Also check this Math.SE post: Relation between cross entropy and conditional entropy.
--
Reference:
$\rm [I]$ Information Theory, Robert B. Ash, Dover Publications, $1990, $ sec. $1.3, $ p. $20.$
