2
$\begingroup$

I've found the explanation that:

The entropy of a binary random (Bernoulli) variable is a function of its probability and maximum when its probability is 0.5 (when it has an entropy of 1 bit). Intuitively, if a measurement is always false (or always true) then we are not uncertain of its value. If instead it is true as often as not, then the uncertainty, and hence the entropy, is maximised.

While I can see how this argument makes sense in the extreme case, I don't intuitively see why this is true. If a binary random variable is 1 with 80% probability and 0 with 20% probability, then when it is 1 will we will not be very surprised. However, when it is 0 we will be quite surprised, and it seems plausible that these two effects could offset each other in some way.

Can anyone provide more intuition for this?

The entropy of a binary random (Bernoulli) variable is a function of its probability and maximum when its probability is 0.5 (when it has an entropy of 1 bit). Intuitively, if a measurement is always false (or always true) then we are not uncertain of its value. If instead it is true as often as not, then the uncertainty, and hence the entropy, is maximised.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ What is the definition of entropy? What value maximizes it? $\endgroup$
    – Sycorax ♦
    Dec 24, 2021 at 6:47

1 Answer 1

Reset to default
4
$\begingroup$
  • The amount of potential information contained in a signal is what we call the entropy, usually denoted by $H$ and defined as follows: $$H(X) = − \sum_X{P(X) \log P(X)}$$
  • The "surprise" is (or you could say proportional to, but it doesn't matter) the inverse of the probability and you can see that the example you mentioned.
  • So the entropy can be written as $$ H(X) = \sum_X{P(X) \log\frac{1}{P(X)}} $$ which is a weighted average of the "surprise" i.e. if the probability is low, then there is a lot of surprise, and consequently a lot of information is conveyed by telling you that X happened.
  • Now to answer the question, YES the surprise is not at maximum when the probability is 0.5 and the event occurs or not (which makes sense since we would be equally surprised by the occurrence and non-occurrence) BUT the entropy/information is a weighted average over the two events
  • Thus, entropy essentially measures the “average surprise” or “average uncertainty” of a random variable. If the distribution $P(X)$ is highly peaked around one value, then we will rarely be surprised by this variable; hence it would be incapable of conveying much information. If on the other hand $P(X)$ is uniformly distributed, then we will be most surprised on average by this variable; hence it could potentially convey a lot of information.
Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Although many texts use your first equation as a definition, my preference is to use the second equation as a definition, for precisely the reason you give: the entropy can then be seen directly as a weighted average. Although it is trivial manipulation that $\log (1/P)$ is (is equal to) $-\log P$ and that the minus sign can be moved, I would be really impressed by anyone who looked at your first equation for the first time and saw immediately that entropy is a weighted average of surprise, defined as here. $\endgroup$
    – Nick Cox
    Dec 24, 2021 at 11:22
  • $\begingroup$ An additional viewpoint that might be helpful: When we speak about the 'surprise', then I like to intuitively regard the surprise as the value $ -\log P(X)$ and the entropy is the expectation value of the surprise $H(X) = E[-\log P(X)]$. You explain this in your third and fourth point but you might expres it as well in formula form to make it more salient. $\endgroup$
    – Sextus Empiricus
    Dec 24, 2021 at 11:37
  • $\begingroup$ In your last sentence the situation is actually reversed. If $P(X)$ is uniformly distributed, then we will be most surprised on average by this variable; hence it conveys less information. $\endgroup$
    – Sextus Empiricus
    Dec 24, 2021 at 11:41
  • $\begingroup$ This is great - thank you! $\endgroup$
    – Fergus
    Jan 4, 2022 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.