Say I can psychically predict the result of a fair coin toss with probability 0.6. On average, how many bits of accurate information am I communicating each time I do it?

I think the answer is

${\large 1-(-\Sigma} p_i \log_2(p_i){\large )} = 1+0.6 \log_2(0.6)+0.4 \log_2(0.4)$

$= 1-0.97095=0.02905$

where the sum is the information entropy function.

The position can be regarded as one where for each throw I receive from my probabilistically more informed vantage point only 0.97095 bits, whereas if you were always to receive the correct answer you would from your point of view receive 1 bit, so from your point of view which is that of a world we assume to be sceptical about psychic abilities it is somehow as if I have produced 1- 0.97095 bits = 0.02905 bits. But is this the right way to go about calculating how many bits I am communicating? Even if it is, I am not sure why it is.


1 Answer 1


Your approach is correct. Similarly, the communicated (gained) bits can be derived by the calculation of Information Gain (IG) in decision trees.

For example, we have $N$ coin observations where 50% is head and 50% is tail. Now, there is a decision rule $d$ (an expert intuition, a psychic power) that can divide them into two $N/2$ groups (left $l$ and right $r$), where there is 60% head in the left and consequently 60% tail in the right (if split was random, we expect 50% in each group). That is, our chance of correctly guessing head (or tail) is increased from 50% to 60% with the help of $d$. We say this decision rule (or psychic power) has an information gain and quantify the gain as the entropy before the split (bits) minus the conditional entropy after the split (informed bits). That is

$$\begin{align*} \text{IG}(d) &= -\sum_{i=1}^{2}p_i\text{log}_2p_i - \left(-\frac{1}{2}\sum_{i=1}^{2}p_i^l\text{log}_2p_i^l -\frac{1}{2}\sum_{i=1}^{2}p_i^r\text{log}_2p_i^r \right)\\ &= -(\frac{1}{2}\text{log}_2\frac{1}{2} + \frac{1}{2}\text{log}_2\frac{1}{2})\\ &\mbox{ }- \left(-\frac{1}{2}(0.6\text{log}_20.6 + 0.4\text{log}_20.4) - \frac{1}{2}(0.4\text{log}_20.4 + 0.6\text{log}_20.6)\right)\\ &= 1 + (0.6\text{log}_20.6 + 0.4\text{log}_20.4) = 0.02905 \end{align*}$$

Therefore, the gain is equivalent to 0.02905 bits (1 bit is lowered to 0.97095 bits). Since the result is independent of $N$, the split process for $N=1$ can be interpreted as changing the probability of guessing the head correctly from 50% to 60% by the psychic power.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.