Shannon Information | Understanding from a Microstate Perspective

Question

So Shannon's information is a way to quantify "distinct knowledge" by means of combination of microstates. So say 1 bit of information in binary system conveys 2 sets of information due to two possible microstates

\begin{pmatrix} 0 \\ 1 \end{pmatrix}

2 bit of information in binary conveys $2^2=4$ set of information due to 4 possible microstates.

\begin{pmatrix} 0\ 0 \\ 0\ 1 \\ 1\ 0 \\ 1\ 1 \end{pmatrix}

Now how to understand this concept using probabilities? If a particular outcome of a random variable say a biased coin flip has a probability of 0.3 for heads then what does it really mean when we say that it conveys $-\log_2(0.5)=1.73 $ bits of information? How does the outcome of the coin has 1.73 microstates?

Answer 1

There is a difference between entropy and number of microstates when dealing with a random process that is not equally probable. In the example of a single coin flip there are only two microstates regardless of coin bias, the coin can come up heads or tail but the entropy will be different for the cases of even coins or biased coins. For an even coin the entropy can be calculated the usual way,

$$H(X) =- \sum_{i \in h,t}p_i \log_2 p_i =- (0.5 \log_2 0.5 + 0.5 \log_2 0.5) = 1\;\mathrm{bits}$$

or because each microstate is equally probable $H(X) = \log_2 2 = 1 \;\mathrm{bits}$

For the biased coin where heads has a probability $p_h = 0.3$ the entropy is,

$$H(X) = - ( 0.3 \log_2 0.3 + 0.7 \log_2 0.7) = 0.88\;\mathrm{bits}$$

The entropy for the biased case is lower because we are less uncertain about the outcome of a coin flip (our intuition tells us that tails is more likely to occur). Another simple example is if we have a random process where we take two coins and flip them then there are four possible microstates $X =\{hh,ht,th,tt\}$

For even coins where each microstate is equally probably the entropy $H(X) = \log_2 4 = 2 \;\mathrm{bits}$

and for two biased coins the entropy is $H(X) = 1.76 \;\mathrm{bits}$ .

Again the entropy of the biased coins is less than the equiprobable case because we know that the coins are weighted towards tails.

Entropy is really tedious to understand because it has use in chemistry, statistical mechanics, and information theory. In my opinion the best and most clear understanding of entropy is "Where We Do Stand on Maximum Entropy"[pg. 12-27] by E.T. Jaynes