Shannon's Entropy interesting problem I found

Question

Here is the problem:

A student is worried that the metro system might not operate properly when he goes home on a given day. There are two reasons for his worries: it is really Cold outside, and they may be working on the construction of metro Extension. He estimates that the risk of failure due to the cold weather is 20%. Independently, they may shut down traffic due to the metro Extension plans with a probability of 50%.

How unpredictable is it that the student may not be able to take the subway home, i.e. that the trains are cancelled, either because of the cold weather or due to the construction work (or both)? Answer in terms of entropy, measured in bits.

The answer:

$$ P(W \lor C) = 1 − P(\neg W)P(\neg C) = 1 − 0.40 = 0.60 \\ ent = −0.6 \log_2(0.6) − 0.4 \log_2(0.4) \approx 0.971 $$

I can understand lets say that 0.6 where it came from and why, but that 0.4 I cannot really what it represents for the student at all.

My approach (which is wrong apparently) is: $ ent = −0.2 \log_2(0.2) − 0.5 \log_2(0.5) −0.1 \log_2(0.1) ≈ 1.28 $, where 0.1 is $0.5 \times 0.2 $ since events are independent.` To my understanding, my expression reads: probability it's cold, or probability there are contractions, or both. What does the correct answer expression read in layman terms?

Answer 1

The definition of entropy is

$$-\sum_{i = 1}^n\left[P(x_i) log(P(x_i))\right]$$

where $x_1, ..., x_n$ are mutually-exclusive events (or symbols, in other interpretations).

In this case, there are two mutually exclusive outcomes: success with probability $P(x_{\mbox{success}}) = 0.4$, and failure with probability $P(x_{\mbox{success}}) = 0.6 = 1 - 0.4$.

Your three-term formula might make sense if there were three mutually-exclusive outcomes: success, failure due to cold but no extension, failure due to extension but not cold, and failure due to both cold and extension. There are a number of problems here:

1. Fundamentally, there is no reason to think that the knowledge of success/failure contains the same amount of information as the knowledge of both success/failure and the exact cause for failure. The entropy of success/failure + reason for failure is greater (in this case) than the entropy of just success/failure.

2. As you can see, for the events to be mutually exclusive, you'd need 4, not 3, events.

3. Your formula has a problem in the "probabilities" not summing to 1. This is just not a valid distribution.