# Confusion about conditional entropy when conditioning on another event

Suppose $$X$$ and $$Y$$ are discrete random variables. I would like to relate the entropy $$H[X \mid Y]$$ to the same entropy conditioned on the additional event $$y>0$$.

My reasoning is as follows: \begin{align} H[ X \mid Y] &= \sum_y p(y) H[ X \mid Y=y] \\ &= \sum_{y >0} p(y) H[ X \mid Y=y, y>0] \\ &+ \sum_{y \le 0} p(y) H[ X \mid Y=y, y\le 0] \end{align} So if I know $$H[ X \mid Y=y, y>0]$$, the above reasoning would allow me to obtain a lower bound for $$H[ X \mid Y]$$ by using $$H[ X \mid Y=y, y>0]$$ assuming I know something about $$p(y>0)$$.

Is my line of thought correct?

If yes, then I'm genuinely puzzled because of the following:

Let $$I$$ be an indicator random variable that is $$1$$ iff $$y>0$$. Now it appears to me that the right-hand side above is equivalent to $$H[ X \mid Y, I]$$. Therefore wouldn't the above imply that $$H[ X \mid Y] = H[ X \mid Y, I]$$ (which isn't true in general)?

• how did you get that the above is equal to H[X∣Y,I] ? Maybe i missed something obvious? Feb 17, 2020 at 13:47
• besides for that, to answer your question: yes. mostly. I don't know what you mean about assuming you know something about p(y>0). H[X|Y] >= the sum over all y> 0 of H[X|Y=y] Feb 17, 2020 at 13:52
• Well $H[ X \mid Y, I] = \sum_{y} p(y,I=1) H[ X \mid Y = y, I=1 ] + \sum_{y} p(y,I=0) H[ X \mid Y = y, I=0 ]$. Isn't this the same as what I got above? What am I missing? Feb 18, 2020 at 5:30

## 1 Answer

Your conclusion that $$H(X|Y)=H(X|YI)$$ in this case is in fact correct, because the indicator variable $$I$$ you have constructed is a function of $$Y$$ alone. In such situations the "data-processing inequality" $$H(X|Y)\geq H(X|YI)$$ for conditional entropy is saturated (one way to look at it is that you can simply apply the data-processing argument in the reverse direction as well, because the pair of random variables $$YI$$ can be generated from $$Y$$ alone).

Another perspective on this is that in general we have a chain rule $$H(X|YI)=H(X|Y)-\mathcal{I}(I:X|Y)$$ (using $$\mathcal{I}$$ for mutual info to avoid notation clash with the indicator variable), but since in your case $$I$$ is a function of $$Y$$ alone, the $$\mathcal{I}(I:X|Y)$$ term is equal to zero (e.g. by noting $$X \leftrightarrow Y \leftrightarrow I$$ form a Markov chain).