# Does it make sense to speak of information content of one bit to be bigger than one bit?

On page 119-120 of Kubat, M.: An Introduction to Machine Learning the following example is given:

Suppose we know that the training examples are labeled as pos or neg, the relative frequencies of these two classes being $p_{pos}$ and $p_{neg}$, respectively. Let us select a random training example. How much information is conveyed by the message, “this example’s class is pos”?

Then the information content is defined as:

$$I_{pos}=-log_2p_{pos}$$

which leads to the following table:

Some values of the information contents (measured in bits) of the message, “this randomly drawn example is positive.” Note that the message is impossible for $p_{pos}=0$

+-------------------+
| p_pos -log2(p_pos)|
+-------------------+
| 1.00   0 bits     |
| 0.50   1 bit      |
| 0.25   2 bits     |
| 0.125  3 bits     |
+-------------------+


My question
Does it make sense to speak of the information content of one bit to be bigger than one bit?

NB
Possibly related but with a different spin: Why am I getting information entropy greater than 1? and Can mutual information gain value be greater than 1

• I'm not sure if this is exactly how to express it, but think of it this way. The information content is related not to the amount of storage required for the bit, but the information that bit represents. So the table you have above can be interpreted as: the more rare an event is, the more information the bit conveys when it is found to be positive. – Arun Jose Sep 30 '16 at 11:25
• @ArunJose: Interesting, could you elaborate and form an answer out of that? Thank you – vonjd Sep 30 '16 at 12:12
• An elaboration of that comment: stats.stackexchange.com/questions/66186/… – kjetil b halvorsen Nov 14 '19 at 0:55