I'm studying for an exam on Bishop's Pattern Recognition and Machine Learning (ISBN: 978- 0387310732). One of the questions in the mock exam paper is:
Three fair coins are flipped sequentially; you win if all three land with the same side up.
i) What is the expected information gain from seeing what side is up in the first coin?
ii) What is the information gain from also seeing the second coin?
I found Bishop's explanation confusing and it is not entirely clear to me what information gain actually represents regardless of its formula.
As far as I have understood the formula for calculated information gain is:
and information gain is the degree of surprise of learning the value of x.
this is not entirely clear to me. My attempt to answer would be:
ii) 0.301 + 0.301
I'm not at all sure about this though. Any help is more than welcome.