# Entropy Conditioned on A Certain Value

Consider this example: I have a box with 10 balls in it, 9 red and 1 blue. I take a ball randomly. Let's call the color $$C$$. If $$C$$ is red, I shout the number zero. If $$C$$ is blue, I roll a fair die and shout the number on the die. Let's have $$X$$ represent the number I shout.

Now, before this experiment, the entropy (uncertainty) of the distribution of $$X$$ is: $$H(X)=-\Sigma_{x=0}^6 P(X=x)\log_2 P(X=x)=-\frac{9}{10}\log_2\frac{9}{10}-6\times\frac{1}{10}\times\frac{1}{6}\log_2\frac{1}{10}\times\frac{1}{6}\approx0.72$$ I can also compute the so-called conditional entropy here, which tells me the expected value of the entropy of $$X$$ if I know $$C$$: $$H(X|C)=P(C=red)H(X|C=red)+P(C=blue)H(X|C=blue)=\\-\frac{9}{10}\times 0-\frac{1}{10}\times\log_2\frac{1}{6}\approx0.25$$ And of course, we can compute the mutual information: $$I(X;C)=H(X)-H(X|C)\approx0.47$$ This means that if we know $$C$$, on average, we know 0.47 bits more about $$X$$ than we used to before knowing $$C$$. But how much we actually would know more (or less, as we see) depends on what $$C$$ turns out to be. So let's say that I take out a ball, and it's blue. Now, I have an amount of uncertainty about the value of $$X$$ that I can compute: $$H(X|C=blue)=-\log_2\frac{1}{6}\approx2.58$$ This basically means that I am more uncertain about $$X$$ after knowing the color of the ball than before it, a.k.a I have less knowledge about what $$X$$ will be! Now, based on what entropy is supposed to represent, this result makes sense. But intuitively, if I know something about the experiment, it should not decrease my level of knowledge about the outcome of the experiment!

So, after all this rambling, here is my question: Is there a more intuitive measure that, in such scenarios, does not decrease? In other words, let's call this measure Knowledge ($$K$$). I want it to have this property (among other intuitive properties): $$K(X)\leq K(X|C=c)$$