1
$\begingroup$

One intuition in the entropy definition is that there is an inverse relationship between the information content of an event and its probability. This makes sense since learning an event which has a small probability gives us more information.

But suppose that we have two coins $A$ and $B$ where for coin $A$, $P(Head)=0.5$ and $P(Tail)=0.5$ and for coin $B$, $P(Head)=0.1$ and $P(Tail)=0.9$. According to Shannon entropy, $H(A)=1$ and $H(B)=0.47$.

In this case I cannot follow this intuition, i.e., why is $H(A)$ larger than $H(B)$? I know that the $log$ used in the entropy formula causes this result but what is the intuition here? According to my intuition, $B$ has one outcome with a small probability and learning it will carry more information than learning any one of the outcomes of $A$. In a sense the probabilities of the outcomes of $B$ balance each other and my intuition says that the information content of $A$ and $B$ should be equal. What is wrong about a formula (different from Shannon entropy formula) which implies H(A) = H(B)?

$\endgroup$
1
1
$\begingroup$

The key distinction to make here is between information content and entropy, which is just the average information content.

Given an event $C$ with probability $p(C)$, the amount of information (in bits) associated with this event, given by its information content, is: $$ \log_2{\frac{1}{p(C)}} $$

The fraction $\frac{1}{p(C)}$ can be thought of as the number that the probability $p(C)$ needs to be multiplied by in order to change it from a probability of $p(C)$ to a probability of $1$. If you are told that event $C$ has occurred, and your prior belief about event $C$ occurring was $p(C)$, then you will change your belief about event $C$ by multiplying it by $\frac{1}{p(C)}$.

Now let $X$ be the result of flipping coin $A$ and let $Y$ be the result of flipping coin $B$, such that $X=1$ if the result of flipping coin $A$ is heads and $Y=1$ if the result of flipping coin $B$ is heads. Now suppose we want to compute the amount of information (or surprisal) associated with observing heads and tails after flipping coin $A$. The amount of information associated with observing heads is: $$ \log_2{\frac{1}{p(X=1)}} = \log_2{\frac{1}{0.5}} = 1 \ \text{bit} $$ And the amount of information associated with observing tails is: $$ \log_2{\frac{1}{p(X=0)}} = \log_2{\frac{1}{0.5}} = 1 \ \text{bit} $$ If we add up these two amounts of information, we get: $$ 1+1=2 \ \text{bits} $$ You can already observe that the total amount of information associated with $X$ is more than its entropy, which is $H(X)=1 \ \text{bit}$, as you've already computed. In fact, the total amount of information associated with $X$ is less than the total amount of information associated with $Y$: $$ \log_2{\frac{1}{p(Y=1)}} + \log_2{\frac{1}{p(Y=0)}} = \log_2{\frac{1}{0.1}} + \log_2{\frac{1}{0.9}} \approx 3.474 \ \text{bits} $$ Which is what your intuition is currently telling you. However, we are not usually interested in the information content given by individual events, but we are instead interested in the information content of a random variable on average.

Suppose that I live on the other side of the world, and I have coins $A$ and $B$ in front of me. I then start flipping each of the coins, and I send you the result of each coin flip consecutively via text message. You then compute the information content of each coin flip as I just did above and then accumulate the results. For example, if I send you the following sequence of results: $$ X = 1, X = 1, Y = 0, Y = 1, Y = 0 $$ Then you would accumulate the information content individually for $X$, denoted $I_X$, and for $Y$, denoted $I_Y$: $$ I_X = \log_2{\frac{1}{p(X=1)}} + \log_2{\frac{1}{p(X=1)}} \\ I_Y = \log_2{\frac{1}{p(Y=0)}} + \log_2{\frac{1}{p(Y=1)}} + \log_2{\frac{1}{p(Y=0)}} $$ However, as I keep sending you these messages, you will soon realize that every time I send you a message, you are only accumulating $I_X$ and $I_Y$ using only four possible numbers: $$ \log_2{\frac{1}{p(X=1)}}, \ \log_2{\frac{1}{p(X=0)}}, \ \log_2{\frac{1}{p(Y=1)}}, \ \log_2{\frac{1}{p(Y=0)}} $$ Therefore, accumulating $I_X$ and $I_Y$ this way is not really interesting to us. Instead, let us compute the averages of $I_X$ and $I_Y$. Suppose I send you a total of $N_X$ messages regarding $X$ (coin $A$) and $N_Y$ messages regarding $Y$ (coin $B$). Then, the averages of $I_X$ and $I_Y$ are: $$ \frac{1}{N_X}I_X = \frac{1}{N_X} \sum_{i=1}^{N_X} \log_2{\frac{1}{p(X=x_i)}} \\ \frac{1}{N_Y}I_Y = \frac{1}{N_Y} \sum_{j=1}^{N_Y} \log_2{\frac{1}{p(Y=y_j)}} $$ Where $x_i$ is the result of the $i^{th}$ flip of coin $A$ and $y_j$ is the result of the $j^{th}$ flip of coin $B$. As we know from the weak law of large numbers: $$ \lim_{N_X\rightarrow\infty} \frac{1}{N_X} \sum_{i=1}^{N_X} \log_2{\frac{1}{p(X=x_i)}} = \mathbb{E}_{p(x)}\left[\log_2{\frac{1}{p(X)}}\right] = H(X) \\ \lim_{N_Y\rightarrow\infty} \frac{1}{N_Y} \sum_{j=1}^{N_Y} \log_2{\frac{1}{p(Y=y_j)}} = \mathbb{E}_{p(y)}\left[\log_2{\frac{1}{p(Y)}}\right] = H(Y) $$ Therefore, as the number of flips of coins $A$ and $B$ increase, the averages of the information content associated with each coin tend to their respective entropies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.