Does it make sense to speak of information content of one bit to be bigger than one bit?

Question

On page 119-120 of Kubat, M.: An Introduction to Machine Learning the following example is given:

Suppose we know that the training examples are labeled as pos or neg, the relative frequencies of these two classes being $p_{pos}$ and $p_{neg}$, respectively. Let us select a random training example. How much information is conveyed by the message, “this example’s class is pos”?

Then the information content is defined as:

$$I_{pos}=-log_2p_{pos}$$

which leads to the following table:

Some values of the information contents (measured in bits) of the message, “this randomly drawn example is positive.” Note that the message is impossible for $p_{pos}=0$

+-------------------+
| p_pos -log2(p_pos)|
+-------------------+
| 1.00   0 bits     |
| 0.50   1 bit      |
| 0.25   2 bits     |
| 0.125  3 bits     |
+-------------------+

My question
Does it make sense to speak of the information content of one bit to be bigger than one bit?

NB
Possibly related but with a different spin: Why am I getting information entropy greater than 1? and Can mutual information gain value be greater than 1

Answer 1

If you look at the formula of information content, what it is a function of is P(pos), or in other words, the probability of finding a 1.

So the information content is not about the bit itself but of the message, and this is encapsulated in its probability of occurrence.

For example, "this apple is orange" has far more information than "this apple is red" because apples are usually red. Encountering an orange apple would mean there is new information. Both these features however can be simple binary dummy variables in your dataset but the latter would be more useful in extracting information.