This is probably a naive question. Why is Entropy formula defined as the way what it is intead of more simpler formula? for example just P(x)*P(y) which I imagine can express uncertainty as well (0.5*0.5 > 0.4*0.6 > 0.3*0.7).
Shannon proved his formula on the basis of following assumptions:
The measure should be continuous — i.e., changing the value of one of the probabilities by a very small amount should only change the entropy by a small amount.
If all the outcomes (ball colours in the example above) are equally likely, then entropy should be maximal. In this case, the entropy increases with the number of outcomes.
- If the outcome is a certainty, then the entropy should be zero.
- The amount of entropy should be the same independently of how the process is regarded as being divided into parts.