2
$\begingroup$

On page 119-120 of Kubat, M.: An Introduction to Machine Learning the following example is given:

Suppose we know that the training examples are labeled as pos or neg, the relative frequencies of these two classes being $p_{pos}$ and $p_{neg}$, respectively. Let us select a random training example. How much information is conveyed by the message, “this example’s class is pos”?

Then the information content is defined as:

$$I_{pos}=-log_2p_{pos}$$

which leads to the following table:

Some values of the information contents (measured in bits) of the message, “this randomly drawn example is positive.” Note that the message is impossible for $p_{pos}=0$

+-------------------+
| p_pos -log2(p_pos)|
+-------------------+
| 1.00   0 bits     |
| 0.50   1 bit      |
| 0.25   2 bits     |
| 0.125  3 bits     |
+-------------------+

My question
Does it make sense to speak of the information content of one bit to be bigger than one bit?

NB
Possibly related but with a different spin: Why am I getting information entropy greater than 1? and Can mutual information gain value be greater than 1

$\endgroup$
  • 2
    $\begingroup$ I'm not sure if this is exactly how to express it, but think of it this way. The information content is related not to the amount of storage required for the bit, but the information that bit represents. So the table you have above can be interpreted as: the more rare an event is, the more information the bit conveys when it is found to be positive. $\endgroup$ – Arun Jose Sep 30 '16 at 11:25
  • $\begingroup$ @ArunJose: Interesting, could you elaborate and form an answer out of that? Thank you $\endgroup$ – vonjd Sep 30 '16 at 12:12
  • $\begingroup$ An elaboration of that comment: stats.stackexchange.com/questions/66186/… $\endgroup$ – kjetil b halvorsen Nov 14 '19 at 0:55
3
$\begingroup$

If you look at the formula of information content, what it is a function of is P(pos), or in other words, the probability of finding a 1.

So the information content is not about the bit itself but of the message, and this is encapsulated in its probability of occurrence.

For example, "this apple is orange" has far more information than "this apple is red" because apples are usually red. Encountering an orange apple would mean there is new information. Both these features however can be simple binary dummy variables in your dataset but the latter would be more useful in extracting information.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.