Let $X$ is a binary variable and $X = \{A, B\}$. The pdf of $X$ is $p(X = A) = .3 \ \ \ p(X=B) = .7$
So it's easy to calculate the entropy of $X$ $$.3 \times \log_2(1/.3) + .7 \times \log_2(1/.7) = 0.88$$

If someone tells me that $X = A$, how much is the information of this message?

Two understandings of information make me confused :

  1. By definition, $I\left(A\right)=-\log \left(.3\right) = 1.20$
  2. Because the X is given now, the entropy of this distribution is reduced to 0. The difference of entropy is $0.88$ which equals to the information of the message.
  • $\begingroup$ $$I\left(A\right)=-\log \left(.3\right) = 1.20$$ where does this definition come from? $\endgroup$ Commented Mar 8, 2022 at 6:30

1 Answer 1


Suppose you have a pack of cards and you are tasked with drawing a card randomly. Entropy is the quantitative measure for the uncertainty associated with guessing that card you picked correctly. For example consider a bag has all red balls and you randomly pick a ball.You are now 100% confident that is 0% uncertain about the outcome (ie- you know it is a red ball). So the entropy or information is 0. The output of entropy is measured in bits.

The entropy function is given by -log(probability of the event). The log function is a mathematical function that takes a smaller value and outputs larger negative values. This is the property that entropy is trying to convey. The function must be able to predict that if there is high uncertainty ,there will be low probability which should predict a high entropy and vice versa. That is what log function does.However it gives negative values which is why you have a -log(p(x) so that the negatives turn to positive entropy values. Hope i was able to help out! Thanks!

p(x) and log (p(x))


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.