This is probably a naive question. Why is Entropy formula defined as the way what it is intead of more simpler formula? for example just P(x)*P(y) which I imagine can express uncertainty as well (0.5*0.5 > 0.4*0.6 > 0.3*0.7).
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Sum of squared probabilities (or its complement, or its reciprocal) is often used, with multiple names such as Gini, Turing, Simpson, Hirschman-Herfindahl, repeat rate, match probability, heterozygosity and (!!!) quadratic entropy. $\endgroup$– Nick CoxCommented Mar 29, 2019 at 8:30
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Shannon proved his formula on the basis of following assumptions:
The measure should be continuous — i.e., changing the value of one of the probabilities by a very small amount should only change the entropy by a small amount.
If all the outcomes (ball colours in the example above) are equally likely, then entropy should be maximal. In this case, the entropy increases with the number of outcomes.
- If the outcome is a certainty, then the entropy should be zero.
- The amount of entropy should be the same independently of how the process is regarded as being divided into parts.
http://folk.uio.no/feder/Fys3130/Entropy/Shannon_entropy.pdf
-
$\begingroup$ For another kind of answer see stats.stackexchange.com/questions/66186/… $\endgroup$ Commented Nov 20, 2022 at 1:49