3
$\begingroup$

For two random variables $X$ and $Y$, the conditional entropy $(Y|X)$ can be given in terms of the joint entropy $(X,Y)$ by:

$H(Y|X) = H(X,Y) − H(X) $

Following page 21 of Cover & Thomas, the Chain Rule gives theorem 2.5.1:

$H(X_1,X_2,...,X_n) = \sum_{i=0}^n H(X_i|X_{i-1},...,X_1) $

Unfortunately, this (and all the other material I have seen) shows the joint entropy in terms of a sum of conditional entropies.

I want to go the other way round, and represent a conditional entropy using only the joint/single entropies.

How do I disentangle the sum term to find e.g.: $ H(Y|X1,X2,X3)$ ?

Is this equivalent to: $H(Y,X1, X2, X3) - H(X1,X2,X3)$ ?

$\endgroup$
4
  • $\begingroup$ The first formula you provide is wrong. The correct is: $H(Y|X)=H(X,Y)-H(X)$ $\endgroup$
    – Andreas G.
    Commented Jan 27, 2018 at 17:01
  • $\begingroup$ Think I spotted and edited that just as you posted, for any future readers who are confused by this comment... :) $\endgroup$
    – Zac
    Commented Jan 27, 2018 at 19:09
  • $\begingroup$ I've noticed that you have edited your post but the formula is still wrong. The correct one is $H(Y|X)=H(X,Y) - H(X)$. What you currently have is $H(Y|X)=H(X) - H(X,Y)$ $\endgroup$
    – Andreas G.
    Commented Jan 27, 2018 at 19:13
  • 1
    $\begingroup$ Whoops; thank you again. Who knew copy and paste could be so hard.. $\endgroup$
    – Zac
    Commented Jan 27, 2018 at 19:15

1 Answer 1

4
$\begingroup$

You can always group random variables together and treat them as single variables.

So if we group $X_1, X_2, X_3$ together, you can do: $$ H(Y,X_1,X_2,X_3)=H(X_1,X_2,X_3) + H(Y|X_1,X_2,X_3) $$

Therefore by rearranging you get: $$ H(Y|X_1,X_2,X_3)=H(Y,X_1,X_2,X_3) - H(X_1,X_2,X_3) $$

which is what you suggested.

I should also note that if you continue to group variables together and keep applying the chain rule for only 2 (groups of) variables you will eventually get the result of theorem 2.5.1

$\endgroup$
1
  • 1
    $\begingroup$ Thank you! Clear answer and the idea of grouping variables into a single variable was new to me and makes utter sense. $\endgroup$
    – Zac
    Commented Jan 27, 2018 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.