# Why doesn't the entropy decrease with an increasing number of observations?

I'm trying to think more about entropy.

I have the following toy example:

Consider a coin flip.

Case 1: I think p_h = 0.5 The entropy of this is 0.5 ln(0.5) x 2 = ln(0.5)

Case 2: I don't know what p_h is and have "no prior" so I use the maximum entropy distribution ie the uniform over p. I then compute the entropy of my p_h, p_t distribution. But since the entropy is again over the heads / tails outcomes the answer is:

$$p_h = \int_0^1 p dp = 0.5$$

So again the distribution is the same.

Case 3: Let's assume I have observed 10 heads and 10 tails so I have a prior of B(11, 11) (ie beta distribution with 20 observations and a uniform prior before the 1st observation). I will again get the same p_h which will give me the same entropy for the ultimate outcome.

If I were to rank each case in terms of "certainty" based on my intuition I would say Case 2 is the most uncertain and Case 1 is the least uncertain since I at least know with certainty the probability of the coin flip.

Why is it that this doesn't show up and gets washed out when considering the entropy of the coin toss.

The motivation for this is that I'll never "know" the probability but I will know the outcome of the coin toss.

• I would suggest writing $log$ rather than $ln$ - in information theory the former can be understood to mean $log_2$ (which gives you the entropy in the common units of bits) but the latter is $log_e$. Nov 26, 2020 at 20:42

More observed flips $$F_{\mathit{obs}}$$ will reduce the uncertainty of your posterior over $$P(p_h | F_{\mathit{obs}})$$. The entropy of the posterior is different from the entropy of the likelihood $$P(f_{\mathit{next}} | p_h^*)$$ for some point estimate $$p_h^*$$. When you commit to a single point estimate, you lose information about your uncertainty about the point estimate and the sample size on which the point estimate was based no longer matters.

## Bayes' rule

To understand the difference between the prior, likelihood, posterior, and the difference between distributions and point estimates, consider Bayes' rule: \begin{align} P( p_H | F_{\mathit{obs}}) = & \frac{P(F_{\mathit{obs}} | p_H) P(p_H)}{P(F_{\mathit{obs}})} \end{align} The $$P(F_{\mathit{obs}} | p_H)$$ term is the likelihood, and reflects the probability of the observed flips assuming that $$p_H$$ has a specific value. The $$P(p_H)$$ term is the prior, and is a probability density over $$p_H$$ itself. Because $$p_H$$ defines a probability distribution, $$P(p_H)$$ is a probability distribution over probability distributions, not a probability distribution over coin flips. The prior expresses how likely each value of $$p_H$$ is before observing any data.

The term on the left $$P(p_H | F_{\mathit{obs}})$$ is the posterior, and, like the prior, is a probability distribution over $$p_H$$, not coin flips. If we pick a uniform prior $$P(p_H) = \text{Beta}(1, 1)$$ and then observe ten heads in twenty flips, $$P(p_H | F_{\mathit{obs}})$$ will say that $$p_H = 0.5$$ has highest posterior density, $$p_H = 0.51$$ has a slightly lower posterior density, and $$p_H = 0.9$$ has a much lower posterior density.

## Distributions versus point estimates

To obtain the probability of seeing heads in the next coin flip given $$F_{\mathit{obs}}$$, called the posterior predictive distribution, we need to use the law of total probability: \begin{align} P(f_{\mathit{next}} = H | F_{\mathit{obs}}) = & \int_\Delta P(f_{\mathit{next}} = H | p_H)P( p_H | F_{\mathit{obs}})dp_H \end{align} where $$P( p_H | F_{\mathit{obs}})$$ is the posterior from Bayes' rule above.

In your example, $$P(f_{\mathit{next}} = H | p_H = 0.5)$$ will have the greatest influence on the probability of the next flip because $$P( p_H | F_{\mathit{obs}})$$ is maximized at $$p_H = 0.5$$, but other possible values of $$p_H$$ will also have some influence.

An alternative is to pick a single "best" point estimate for $$p_H$$, such as the maximum a-posteriori estimate: \begin{align} p_H^* = &~ \underset{p_H}{\arg\max}~ P( p_H | F_{\mathit{obs}}) \end{align} and then simply use the point estimate: \begin{align} P(f_{\mathit{next}} = H | F_{\mathit{obs}}) \approx &~ P(f_{\mathit{next}} = H | p_H^*) \end{align} This approximation can be viewed as the limit of replacing $$P( p_H | F_{\mathit{obs}})$$ in the posterior predictive distribution with distributions that are increasingly concentrated on $$p_H^*$$. Because it is an approximation, it loses information compared to the true posterior $$P( p_H | F_{\mathit{obs}})$$.

Concretely, you could not possibly have observed 51% heads in twenty flips but you could have in 100 flips. The full posterior appreciates this by giving lower posterior density to $$p_H = 0.51$$ when there are 50 heads in 100 flips than when there are ten heads in twenty flips, but taking a point estimate completely ignores the possibility that $$p_H = 0.51$$ in both cases.

## Differential entropy of the posterior

We can consider further the uncertainty of the posterior. The (differential) entropy of this posterior is: \begin{align} H(P(p_H | \alpha_H, \alpha_T)) = & -\int_\Delta P(p_H | \alpha_H, \alpha_T) \ln P(p_H | \alpha_H, \alpha_T) dp_h \\ = &~\ln B(\alpha_H, \alpha_T) - (\alpha_H - 1)\psi(\alpha_H) - (\alpha_T - 1)\psi(\alpha_T) + \\ &~(\alpha_H + \alpha_T - 2)\psi(\alpha_H + \alpha_T) \end{align} Where $$B$$ is the Beta function and $$\psi$$ is the derivative of the log gamma function (also called the Digamma function). To understand how the differential entropy changes with observations, let's plot the entropy of a symmetric Beta distribution as the sum of the shape parameters increases. We see that entropy is maximal for a symmetric Beta distribution at the blue line, where both shape parameters are equal to one. In R:

symmetric_beta_entropy <- function(size) {
a <- size / 2
b <- size / 2
log(beta(a, b)) - (a - 1) * digamma(a) - (b - 1) * digamma(b) +
(a + b - 2) * digamma(a + b)
}

library(ggplot2)
ggplot(data.frame(N=0), aes(x=N)) +
theme_bw() +
stat_function(fun=symmetric_beta_entropy) +
xlim(1, 20) +
geom_vline(xintercept=2, color='blue') +
ylab("H(P(p_h | N/2, N/2))")


• thank you for the answer to follow up; just to understand by $P(p_H | F_obs)$ is the probability of getting heads given the observations right? This is just the Beta distribution assuming you have a uniform prior (ie a B(1,1))? I agree that the entropy of this would decrease as the number of observations increases but I'm trying to understand if the uncertainty of the the outcome of heads vs tails would change at all or not - my calculations indicate no but that seems like a weird (or incorrect result) Nov 26, 2020 at 19:27
• also I disagree that committing to a single point estimate is losing information I would say that this is the most informed I can be since it is equivalent to a $lim_{x->\inf} Beta(x, x)$ ie i've seen an "infinite" amount of coin tosses Nov 26, 2020 at 19:29
• I've added more text on the difference between distributions over $p_H$ and distributions over flips, and the difference between using posteriors and taking point estimates. Pretending you have seen an infinite amount of coin tosses loses the information that you have not actually seen an infinite amount of coin tosses. For further study, I recommend searching for "Beta-Binomial" model on this site or elsewhere. Nov 27, 2020 at 9:11