How to apply the formula for Shannon entropy to a 4-sided die? I have come across calculating entropy, via the formula:
\begin{equation}
Entropy(p) = -\sum_{i=1} ^{N}p_i \log_2(p_i)
\end{equation}
Referring to this formula, how would I calculate the entropy of a fair 4 sided die for example?
I'm presuming 1/4 will be 'p', and 'i' will be the index?
 A: The distribution of each outcome would be $[1/4,1/4/,1/4,1/4]$, i.e. $p_i=1/4$ for each $i\in \{1,2,3,4\}$. This yields $$H(p)=-4\times \frac{1}{4}\times\log_2\frac{1}{4}=2$$
As pointed out in the comments, the answer for an N-sided fair die would be $\log N$:
$$H(p)=-\sum_{i=1}^N\frac{1}{N} \log\frac{1}{N}=-\log \frac{1}{N}=\log N$$
This is true for log with any base, depending on the entropy definition used. In this particular problem, it is base $2$.
A: I think it's worth mentioning that not only does the formula reduce to $\log N$ ($\log_2N$ if we're specifically talking about entropy measured in bits) in the case of $N$ different equally likely possibilities, but the formula for different probabilities is derived from the equally likely possibilities case, rather than the other way around. Shannon entropy is the number of bits it takes, on average, to specify a state. If all the states are equally likely, we can drop the "on average" part. Then we have that $S$ bits can specify $2^S$ states, or $\log_2 N$ bits are required to specify $N$ states. Specifying $4$ different states can clearly be done with $2$ bits; there are $4$ different $2$-digit binary numbers.
As for cases where the probabilities are different, suppose we have one state with $p=0.5$ and two states with $p = 0.25$. Then we can assign the first state the label $0$ and the other two states $10$ and $11$. Half the time we're using $1$ bit, and half the time we're using $2$ bits, so we have $1.5$ bits of entropy. It's more complicated when the probabilities aren't powers of $2$, but we can still give a similar calculation in the limit. That is, if we have a string of length $l$ consisting of independently random letters, and individual letters have an entropy of $S$, then we can on average represent the string with less than $S(l+1)$ bits, and the bits per letter goes to $S$ in the limit.
A: The sum is over all possible outcomes, and $p_i$ are the probabilities for each. There are 4 outcomes, so the sum will run over each one of those - that is, $i$ will stand for a face of the die, and what will go in for $p_i$ will be the probability for the die to land with that face sticking up. For a fair die, that will be $\frac{1}{4}$ and then you will be right, the entropy will be $4 \left(-\frac{1}{4} \lg \frac{1}{4}\right) = \lg 4$ or 2 bits.
