# Does it make sense to speak of information content of one bit to be bigger than one bit?

On page 119-120 of Kubat, M.: An Introduction to Machine Learning the following example is given:

Suppose we know that the training examples are labeled as pos or neg, the relative frequencies of these two classes being $p_{pos}$ and $p_{neg}$, respectively. Let us select a random training example. How much information is conveyed by the message, “this example’s class is pos”?

Then the information content is defined as:

$$I_{pos}=-log_2p_{pos}$$

which leads to the following table:

Some values of the information contents (measured in bits) of the message, “this randomly drawn example is positive.” Note that the message is impossible for $p_{pos}=0$

+-------------------+
| p_pos -log2(p_pos)|
+-------------------+
| 1.00   0 bits     |
| 0.50   1 bit      |
| 0.25   2 bits     |
| 0.125  3 bits     |
+-------------------+


My question
Does it make sense to speak of the information content of one bit to be bigger than one bit?

NB
Possibly related but with a different spin: Why am I getting information entropy greater than 1? and Can mutual information gain value be greater than 1

• I'm not sure if this is exactly how to express it, but think of it this way. The information content is related not to the amount of storage required for the bit, but the information that bit represents. So the table you have above can be interpreted as: the more rare an event is, the more information the bit conveys when it is found to be positive. Sep 30, 2016 at 11:25
• @ArunJose: Interesting, could you elaborate and form an answer out of that? Thank you Sep 30, 2016 at 12:12
• An elaboration of that comment: stats.stackexchange.com/questions/66186/… Nov 14, 2019 at 0:55