I'm studying for an exam on Bishop's Pattern Recognition and Machine Learning (ISBN: 978- 0387310732). One of the questions in the mock exam paper is:

Three fair coins are flipped sequentially; you win if all three land with the same side up.

i) What is the expected information gain from seeing what side is up in the first coin?

ii) What is the information gain from also seeing the second coin?

I found Bishop's explanation confusing and it is not entirely clear to me what information gain actually represents regardless of its formula.

As far as I have understood the formula for calculated information gain is:


and information gain is the degree of surprise of learning the value of x.

this is not entirely clear to me. My attempt to answer would be:

i) 0.301

ii) 0.301 + 0.301

I'm not at all sure about this though. Any help is more than welcome.

Thank you

  • $\begingroup$ Please add the self-study tag and read its tag-wiki, modifying your question to follow the guidelines there. $\endgroup$ – Glen_b May 3 '15 at 12:15

The information of an event is defined as: ${I} (p)=\log(1/p)$ The probability of the first coin showing head is 0,5. Therefore the information gain is 0,301.

The information gain of the second event can be computed using the probability of the second event 0,25, therefore the information gain is 0,602.

Therefore, yes, you are right.

  • 2
    $\begingroup$ Just note though that while the base of the log is technically arbitrary, certain bases are more common than others, or more typical for certain settings. You seem to be using base 10. This is fine, but base 2 would be more natural for a coin flip. In this base, the answers are simply "1" and "2". Another note: from the phrasing of the second question it's not entirely clear whether we're asked about the information gained by observing the second coin flip, or by observing the first two flips. In the first case, the answer to ii) is "1", while in the second, it's "2" (again in base 2). $\endgroup$ – Ruben van Bergen Jan 19 '17 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.