Consider this example: I have a box with 10 balls in it, 9 red and 1 blue. I take a ball randomly. Let's call the color $C$. If $C$ is red, I shout the number zero. If $C$ is blue, I roll a fair die and shout the number on the die. Let's have $X$ represent the number I shout.

Now, before this experiment, the entropy (uncertainty) of the distribution of $X$ is: $$H(X)=-\Sigma_{x=0}^6 P(X=x)\log_2 P(X=x)=-\frac{9}{10}\log_2\frac{9}{10}-6\times\frac{1}{10}\times\frac{1}{6}\log_2\frac{1}{10}\times\frac{1}{6}\approx0.72$$ I can also compute the so-called conditional entropy here, which tells me the expected value of the entropy of $X$ if I know $C$: $$H(X|C)=P(C=red)H(X|C=red)+P(C=blue)H(X|C=blue)=\\-\frac{9}{10}\times 0-\frac{1}{10}\times\log_2\frac{1}{6}\approx0.25$$ And of course, we can compute the mutual information: $$I(X;C)=H(X)-H(X|C)\approx0.47$$ This means that if we know $C$, on average, we know 0.47 bits more about $X$ than we used to before knowing $C$. But how much we actually would know more (or less, as we see) depends on what $C$ turns out to be. So let's say that I take out a ball, and it's blue. Now, I have an amount of uncertainty about the value of $X$ that I can compute: $$H(X|C=blue)=-\log_2\frac{1}{6}\approx2.58$$ This basically means that I am more uncertain about $X$ after knowing the color of the ball than before it, a.k.a I have less knowledge about what $X$ will be! Now, based on what entropy is supposed to represent, this result makes sense. But intuitively, if I know something about the experiment, it should not decrease my level of knowledge about the outcome of the experiment!

So, after all this rambling, here is my question: Is there a more intuitive measure that, in such scenarios, does not decrease? In other words, let's call this measure Knowledge ($K$). I want it to have this property (among other intuitive properties): $$K(X)\leq K(X|C=c)$$


1 Answer 1


Now, based on what entropy is supposed to represent, this result makes sense. But intuitively, if I know something about the experiment, it should not decrease my level of knowledge about the outcome of the experiment!

Are you sure that the latter is intuitive? I don't think it is, and I can think of many scenarios where additional information would make us more uncertain about the outcome of an experiment. Your balls-and-dice gives one example where additional knowledge in an experiment makes the outcome more uncertain, but let's try to find a more intuitive example from real life.

Suppose you are friends with a married couple with a child and you want to make an inference of who is the biological father of the child. (Let's stipulate that the couple have been married since prior to the conception of the child.) In the ordinary case, not having any specific information to the contrary, you would be highly confident that the husband is the biological father of the child --- that is a highly likely outcome and so the entropy of the "who is the biological father" random variable is low. However, suppose you are having beers with this guy and he lets you know that he just did a paternity test that showed that he is not the biological father of the child (he's devastated, so it's your shout). You've just been given more information than you had before, but this new information makes the identity of the biological father much more uncertain --- i.e., the entropy has increased substantially conditional on this event.

In fact, this type of situation is pretty similar to what you are modelling in your balls-and-dice scenario. We have one highly likely event that leads to a specific outcome for the random variable under analysis, but we also have a contrary low-probability event that leads to a highly variable outcome for the random variable under analysis. Taking your model as the analogy would be like saying that there is a 90% chance that the husband is the biological father of the child, and a 10% chance that it was one of six other guys, but once you condition on the outcome of a paternity test, you go to one branch or the other.

So far we have just two examples, but you can see that the general pattern here is that we have a single high-probability outcome which makes the entropy low, but we have a set of remaining low-probability outcomes that are fairly evenly spread across a number of outcomes. Consequently, we start with low entropy, but when we condition on the negation of the high-probability outcome, we get a high conditional entropy for the remaining outcomes. This situation is so common in day-to-day life that we have a saying for it in standard English vernacular --- we say that the new information "raises more questions than answers".

(To answer your final question: I think it is wrong to look for a measure of uncertainty with this property, precisely because it does not allow you to describe these common scenarios where a new piece of information makes you more uncertain about an outcome than you were before. The property you put forward is not an intuitive property in my view, and it would tend to destroy the measure.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.