How the entropies of the random variables in conditional entropy affect its value? We know that the conditional entropy $H(Y|X)\to 0$ as $X$ determines the value of $Y$. Now, I have the intuition of that $H(Y|X)<H(Y|Z)$ if $H(X)<H(Z)$. With this, and from the first statement, I interpret that a random variable whose entropy is smaller ($X$) provides more knowledge on the variable ($Y$) than other variable whose entropy is larger ($Z$), assuming that $P(Y,X)\neq P(Y)P(X)$ and $P(Y,Z)\neq P(Y)P(Z)$. Is that correct? I'm not sure how to verify it, please help me.
As a preliminary research, I suspect that I need to analyze $P(Y,\cdot)/P(\cdot)$ in the definition of conditional entropy, but I'm not sure how to do it properly, in the case it makes sense.
 A: No, this isn't true—your 'intuition' statement is incorrect. It assumes that the information that a variable provides about another is proportional to the amount of information in the variable itself. It could be that the higher-entropy variable is more related.

Let's demonstrate with an example.
Consider 5 independent random variables with the same entropy: $A$, $B$, $C$, $D$, and $E$, and a sixth $F$ with greater entropy. (Perhaps they all are uniform on the same support, except $F$ has a larger support.)
Now define $Y \triangleq (A, B, C)$, $X \triangleq (A, D, E)$, and $Z \triangleq (A, B, F)$. As the joint entropy of independent variables is their entropies' sum, $H(Y) = H(A) + H(B) + H(C)$. Similarly, $H(X) = H(A) + H(D) + H(E)$, and $H(Z) = H(A) + H(B) + H(E) > H(X)$.





A
B
C
D
E
F (bigger!)




Y
✓
✓
✓





X
✓


✓
✓



Z
✓
✓



✓




So far, we've shown your precondition $H(X) < H(Z)$ by the construction of our variables. Now what about the conditional entropy?
$$
\begin{align}
H(Y \mid X) &= H(B) + H(C) \\
H(Y \mid Z) &= \hphantom{H(B) + } H(C)
\end{align}
$$
Because we know $H(B) = H(C)$, this means that $Z$ reduced our uncertainty about $Y$ more than $X$ did! And it did this despite having larger entropy.

Note that $A$ wasn't strictly necessary here. If we cut it out of the example, then $X$ would still have smaller entropy than $Z$, but it would have no contribution to $Y$ at all. I think it's more general to keep it in.


Please, can you tell why it holds that (|)=()+()?

Sure thing!
It is always true for any $J$ and $K$ that $H(J, K) = H(J \mid K) + H(K)$. This is the chain rule for entropy, which mirrors the chain rule for probability. It says that knowledge of $K$ reduces our uncertainty about $(J, K)$ by the amount $H(K)$; the remaining uncertainty is $H(J \mid K)$.
In our case, we're curious about $H(Y \mid X)$. Let's figure it out from the joint entropy $H(Y, X)$ and the entropy of $X$, $H(X)$.
Fortunately, from here it's just algebra. On the third line, we rely on the fact that independent variables' entropies add.
$$
\begin{align}
H(Y, X) &= H(Y \mid X) + H(X) \\
H(A, B, C, D, E) &= H(Y \mid X) + H(A, D, E) \\
H(A) + H(B) + H(C) + H(D) + H(E) &= H(Y \mid X) + H(A) + H(D) + H(E) \\ 
H(B) + H(C) &= H(Y \mid X) \\ 
\end{align}
$$
If that seemed a bit long-winded, the intuitive way to think about it is this: If we know $X$, then we know that much about $Y$. The only missing information about $Y$ is in $B$ and $C$.
