Why do we weight the surprisals by the probabilities when computing the entropy?

Question

Let's say that a random variable $X$ takes the values $(x_1, x_2, \dots, x_n)$ over the sample space $\Omega$.

As far as my understanding goes, the entropy of a given variable is meant to give an idea of how small are the probabilities of the same variable taking the values that it takes, and thus knowledge of a variable that has a high value of entropy would significantly limit the possible values taken by other variables over the same sample space.

My question is, if we want to measure how small, on average, the probabilities $(p(x_i))_{i \in [1,n]}$ are, why don't we just calculate the result of the term:

$$\frac{1}{n}\sum_{i \in [1, n] } log(\frac{1}{p(x_i)})$$

Why do we have to weight it by the probabilities $(p(x_i))_{i \in [1, n]}$ which seem to produce a counter effect to the one we intend to measure (the higher the surprisal the smaller its weigh and thus the less important it is to the overall sum)?

I can see that the textbook formula for entropy (for lack of a better term) is indeed the average of the variable $-log(P(X))$ over the sample space $\Omega$. But it seems more intuitive to me to calculate the expectation of the same variable over $X(\Omega)$ instead, which translates to the formula aformentioned.

Answer 1

First of all, intuitively, it is an expected value, because we want to know what is the "average uncertainty" of a random variable. It also tells you about the "number of bits on average required to describe the random variable". If you didn't weigh the probabilities, the "average" wouldn't make sense. Let's re-write the entropy as an expected value of the information content of the $x_i$ values

$$ H(X) = E[I(X)] = \sum_i p(x_i) I(x_i) $$

With your formula, this would be just a (normalized) sum $\sum_i I(x_i)$, why would you care about the sum? You don't care about the "total information content" but the average. If you get a random message $x_i$, you can use the expected value to make a rough guess on how many bits you need to encode it. The probability that you observe a particular $x_i$ value is $p(x_i)$, so that should be its contribution to the average. It is just an expected value of the function (that tells us about the information content) of a random variable.

But Shannon (1948) also listed the formal requirements he wanted the entropy to follow

1. $H$ should be continuous in the $p_i$.
2. If all the $p_i$ are equal, $p_i = \frac{1}{n}$, then $H$ should be a monotonic increasing function of $n$. With equally likely events there is more choice, or uncertainty, when there are more possible events.
3. If a choice be broken down into two successive choices, the original $H$ should be the weighted sum of the individual values of $H$.

and, as he proves, only the expected value satisfies them.