# How to represent conditional entropy in terms of joint entropy?

For two random variables $X$ and $Y$, the conditional entropy $(Y|X)$ can be given in terms of the joint entropy $(X,Y)$ by:

$H(Y|X) = H(X,Y) − H(X)$

Following page 21 of Cover & Thomas, the Chain Rule gives theorem 2.5.1:

$H(X_1,X_2,...,X_n) = \sum_{i=0}^n H(X_i|X_{i-1},...,X_1)$

Unfortunately, this (and all the other material I have seen) shows the joint entropy in terms of a sum of conditional entropies.

I want to go the other way round, and represent a conditional entropy using only the joint/single entropies.

How do I disentangle the sum term to find e.g.: $H(Y|X1,X2,X3)$ ?

Is this equivalent to: $H(Y,X1, X2, X3) - H(X1,X2,X3)$ ?

• The first formula you provide is wrong. The correct is: $H(Y|X)=H(X,Y)-H(X)$ – Andreas G. Jan 27 '18 at 17:01
• Think I spotted and edited that just as you posted, for any future readers who are confused by this comment... :) – Zac Jan 27 '18 at 19:09
• I've noticed that you have edited your post but the formula is still wrong. The correct one is $H(Y|X)=H(X,Y) - H(X)$. What you currently have is $H(Y|X)=H(X) - H(X,Y)$ – Andreas G. Jan 27 '18 at 19:13
• Whoops; thank you again. Who knew copy and paste could be so hard.. – Zac Jan 27 '18 at 19:15

So if we group $X_1, X_2, X_3$ together, you can do: $$H(Y,X_1,X_2,X_3)=H(X_1,X_2,X_3) + H(Y|X_1,X_2,X_3)$$
Therefore by rearranging you get: $$H(Y|X_1,X_2,X_3)=H(Y,X_1,X_2,X_3) - H(X_1,X_2,X_3)$$