For two random variables $X$ and $Y$, the conditional entropy $(Y|X)$ can be given in terms of the joint entropy $(X,Y)$ by:

$H(Y|X) = H(X,Y) − H(X) $

Following page 21 of Cover & Thomas, the Chain Rule gives theorem 2.5.1:

$H(X_1,X_2,...,X_n) = \sum_{i=0}^n H(X_i|X_{i-1},...,X_1) $

Unfortunately, this (and all the other material I have seen) shows the joint entropy in terms of a sum of conditional entropies.

I want to go the other way round, and represent a conditional entropy using only the joint/single entropies.

How do I disentangle the sum term to find e.g.: $ H(Y|X1,X2,X3)$ ?

Is this equivalent to: $H(Y,X1, X2, X3) - H(X1,X2,X3)$ ?

  • $\begingroup$ The first formula you provide is wrong. The correct is: $H(Y|X)=H(X,Y)-H(X)$ $\endgroup$ – Andreas G. Jan 27 '18 at 17:01
  • $\begingroup$ Think I spotted and edited that just as you posted, for any future readers who are confused by this comment... :) $\endgroup$ – Zac Jan 27 '18 at 19:09
  • $\begingroup$ I've noticed that you have edited your post but the formula is still wrong. The correct one is $H(Y|X)=H(X,Y) - H(X)$. What you currently have is $H(Y|X)=H(X) - H(X,Y)$ $\endgroup$ – Andreas G. Jan 27 '18 at 19:13
  • 1
    $\begingroup$ Whoops; thank you again. Who knew copy and paste could be so hard.. $\endgroup$ – Zac Jan 27 '18 at 19:15

You can always group random variables together and treat them as single variables.

So if we group $X_1, X_2, X_3$ together, you can do: $$ H(Y,X_1,X_2,X_3)=H(X_1,X_2,X_3) + H(Y|X_1,X_2,X_3) $$

Therefore by rearranging you get: $$ H(Y|X_1,X_2,X_3)=H(Y,X_1,X_2,X_3) - H(X_1,X_2,X_3) $$

which is what you suggested.

I should also note that if you continue to group variables together and keep applying the chain rule for only 2 (groups of) variables you will eventually get the result of theorem 2.5.1

  • 1
    $\begingroup$ Thank you! Clear answer and the idea of grouping variables into a single variable was new to me and makes utter sense. $\endgroup$ – Zac Jan 27 '18 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.