2
$\begingroup$

We know that the conditional entropy $H(Y|X)\to 0$ as $X$ determines the value of $Y$. Now, I have the intuition of that $H(Y|X)<H(Y|Z)$ if $H(X)<H(Z)$. With this, and from the first statement, I interpret that a random variable whose entropy is smaller ($X$) provides more knowledge on the variable ($Y$) than other variable whose entropy is larger ($Z$), assuming that $P(Y,X)\neq P(Y)P(X)$ and $P(Y,Z)\neq P(Y)P(Z)$. Is that correct? I'm not sure how to verify it, please help me.

As a preliminary research, I suspect that I need to analyze $P(Y,\cdot)/P(\cdot)$ in the definition of conditional entropy, but I'm not sure how to do it properly, in the case it makes sense.

$\endgroup$
0

1 Answer 1

3
$\begingroup$

No, this isn't true—your 'intuition' statement is incorrect. It assumes that the information that a variable provides about another is proportional to the amount of information in the variable itself. It could be that the higher-entropy variable is more related.


Let's demonstrate with an example.

Consider 5 independent random variables with the same entropy: $A$, $B$, $C$, $D$, and $E$, and a sixth $F$ with greater entropy. (Perhaps they all are uniform on the same support, except $F$ has a larger support.)

Now define $Y \triangleq (A, B, C)$, $X \triangleq (A, D, E)$, and $Z \triangleq (A, B, F)$. As the joint entropy of independent variables is their entropies' sum, $H(Y) = H(A) + H(B) + H(C)$. Similarly, $H(X) = H(A) + H(D) + H(E)$, and $H(Z) = H(A) + H(B) + H(E) > H(X)$.

A B C D E F (bigger!)
Y
X
Z

So far, we've shown your precondition $H(X) < H(Z)$ by the construction of our variables. Now what about the conditional entropy?

$$ \begin{align} H(Y \mid X) &= H(B) + H(C) \\ H(Y \mid Z) &= \hphantom{H(B) + } H(C) \end{align} $$

Because we know $H(B) = H(C)$, this means that $Z$ reduced our uncertainty about $Y$ more than $X$ did! And it did this despite having larger entropy.


Note that $A$ wasn't strictly necessary here. If we cut it out of the example, then $X$ would still have smaller entropy than $Z$, but it would have no contribution to $Y$ at all. I think it's more general to keep it in.


Please, can you tell why it holds that 𝐻(𝑌|𝑋)=𝐻(𝐵)+𝐻(𝐶)?

Sure thing!

It is always true for any $J$ and $K$ that $H(J, K) = H(J \mid K) + H(K)$. This is the chain rule for entropy, which mirrors the chain rule for probability. It says that knowledge of $K$ reduces our uncertainty about $(J, K)$ by the amount $H(K)$; the remaining uncertainty is $H(J \mid K)$.

In our case, we're curious about $H(Y \mid X)$. Let's figure it out from the joint entropy $H(Y, X)$ and the entropy of $X$, $H(X)$.

Fortunately, from here it's just algebra. On the third line, we rely on the fact that independent variables' entropies add.

$$ \begin{align} H(Y, X) &= H(Y \mid X) + H(X) \\ H(A, B, C, D, E) &= H(Y \mid X) + H(A, D, E) \\ H(A) + H(B) + H(C) + H(D) + H(E) &= H(Y \mid X) + H(A) + H(D) + H(E) \\ H(B) + H(C) &= H(Y \mid X) \\ \end{align} $$

If that seemed a bit long-winded, the intuitive way to think about it is this: If we know $X$, then we know that much about $Y$. The only missing information about $Y$ is in $B$ and $C$.

$\endgroup$
4
  • $\begingroup$ Dear Arya McCarthy, Thanks for your reply. Please, can you tell why it holds that $H(Y|X)=H(B)+H(C)$? $\endgroup$
    – Nacho
    Mar 10, 2021 at 22:37
  • 1
    $\begingroup$ Happy to! I just updated the answer to include that information at the bottom. $\endgroup$ Mar 11, 2021 at 15:03
  • $\begingroup$ Thank you very much @Arya McCarthy. Your explanation is very complete. Now, let me take advantage of your good knowledge of this matter to ask for your opinion. What statistical properties do you think that $Z$ should have to reach out to $H(Y|Z) < H(Y|X) $? $\endgroup$
    – Nacho
    Mar 11, 2021 at 19:28
  • 1
    $\begingroup$ Put simply, $Z$ should be more related to $Y$ than $X$ is. For anything beyond this toy example, finding a fitting $Z$ here may require domain knowledge. It's not about statistical properties of $Z$ by itself! ...This is a bit of circular reasoning, but the mutual information $I(Y, Z)$ should be greater than $I(Y, X)$. $\endgroup$ Mar 11, 2021 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.