Suppose $X$ and $Y$ are discrete random variables. I would like to relate the entropy $H[X \mid Y]$ to the same entropy conditioned on the additional event $y>0$.

My reasoning is as follows: \begin{align} H[ X \mid Y] &= \sum_y p(y) H[ X \mid Y=y] \\ &= \sum_{y >0} p(y) H[ X \mid Y=y, y>0] \\ &+ \sum_{y \le 0} p(y) H[ X \mid Y=y, y\le 0] \end{align} So if I know $H[ X \mid Y=y, y>0]$, the above reasoning would allow me to obtain a lower bound for $H[ X \mid Y]$ by using $H[ X \mid Y=y, y>0]$ assuming I know something about $p(y>0)$.

Is my line of thought correct?

If yes, then I'm genuinely puzzled because of the following:

Let $I$ be an indicator random variable that is $1$ iff $y>0$. Now it appears to me that the right-hand side above is equivalent to $H[ X \mid Y, I]$. Therefore wouldn't the above imply that $H[ X \mid Y] = H[ X \mid Y, I]$ (which isn't true in general)?

  • $\begingroup$ how did you get that the above is equal to H[X∣Y,I] ? Maybe i missed something obvious? $\endgroup$ Feb 17, 2020 at 13:47
  • $\begingroup$ besides for that, to answer your question: yes. mostly. I don't know what you mean about assuming you know something about p(y>0). H[X|Y] >= the sum over all y> 0 of H[X|Y=y] $\endgroup$ Feb 17, 2020 at 13:52
  • $\begingroup$ Well $H[ X \mid Y, I] = \sum_{y} p(y,I=1) H[ X \mid Y = y, I=1 ] + \sum_{y} p(y,I=0) H[ X \mid Y = y, I=0 ]$. Isn't this the same as what I got above? What am I missing? $\endgroup$
    – csstudent
    Feb 18, 2020 at 5:30

1 Answer 1


Your conclusion that $H(X|Y)=H(X|YI)$ in this case is in fact correct, because the indicator variable $I$ you have constructed is a function of $Y$ alone. In such situations the "data-processing inequality" $H(X|Y)\geq H(X|YI)$ for conditional entropy is saturated (one way to look at it is that you can simply apply the data-processing argument in the reverse direction as well, because the pair of random variables $YI$ can be generated from $Y$ alone).

Another perspective on this is that in general we have a chain rule $H(X|YI)=H(X|Y)-\mathcal{I}(I:X|Y)$ (using $\mathcal{I}$ for mutual info to avoid notation clash with the indicator variable), but since in your case $I$ is a function of $Y$ alone, the $\mathcal{I}(I:X|Y)$ term is equal to zero (e.g. by noting $X \leftrightarrow Y \leftrightarrow I$ form a Markov chain).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.