Why doesn't estimating Shannon entropy with a histogram converge to its true value?

Question

I'm following the third recipe of this answer to estimate the Shannon entropy of my samples using histograms. My expectation was, increasing the sample size should lead to a better estimation of the true entropy. To test that, I sampled from a Gaussian distribution $N(0, \sigma^2)$ with known entropy $H(X) = 0.5[1 + \log(2\pi \sigma^2)]$:

import numpy as np
from scipy.stats import entropy # by default in natural log
import matplotlib.pyplot as plt

sigma = 2
H0 = 0.5*(1 + np.log(2*np.pi*sigma**2)) # theoretical value in nats

Hs = []
ns = np.logspace(2,6,5, dtype=int)
for n in ns:
    X = np.random.normal(0, sigma, size=n)
    nbins = int(n/20)  # <--- This is causes divergence, read UPDATE 2
    hist, bin_edges = np.histogram(X, nbins, density=True)
    Hs.append(entropy(hist)) # will be automatically normalized to sum to 1

plt.plot(ns, Hs, '-x', label='estimation')
plt.hlines(H0, ns[0], ns[-1], linestyle='--', color='k', label='exact H')
plt.legend()
plt.yticks(np.arange(11))
plt.xscale('log')
plt.xlabel('num samples')
plt.ylabel('sample entropy')

To my surprise, they diverge:

Am I missing something very simple?

I found a relevant question that is concerned about the case where the probability distribution is either "peaked" or "flat". The upshot of the most voted answer is that one has to normalize the entropy by $\log n$ where $n$ is the number of samples. Although I find this normalization sensible, I still observe an offset between the theoretical value and the estimation:

plt.plot(ns, np.array(Hs)/np.log(ns), '-x', label='normalized estimation') # normalized

What causes this discrepancy?

As @sextus-empiricus noted below, scaling the number bins with the sample size leads to a coarse density estimation. In fact, fixing the number of bins, resolves the divergence. However, no matter what number of bins one chooses, there's still some offset between the estimated (normalized or non-normalized) entropy and the theoretical value, as you can see below:

What is the reason behind this gap? The curve displacement by nbins clearly suggest that there's a relationship between the two, but I found the reason elusive.

Answer 1

Here is an image that I made with the R-code below. The computation is with n-bins = 50.

There is in this simulation also an offset but smaller and with more bins I get a smaller offset. This might be because you approach the values of a discretized normal distribution and not the values of a continuous normal distribution. So one should not expect the same.

For small sample sizes there is lower entropy. This might be counter intuitive because one would imagine a small sample to be more variable and more chaotic. But, the state described by the histogram of a small sample is a very particular state and has more order than the average of many different samples. The increase of the sample size can be seen as blending multiple states and that creates an average that blends out to a smooth histogram, a histogram that looks like a more homogeneous sample.

set.seed(1)
n_range = 10^seq(2,6,0.2)
entropy_range = c()
n_bins = 50

for (n in n_range) {
  x = rnorm(n, mean = 0, sd = 2)
  h = hist(x, breaks = n_bins, freq = 0)
  p = h$density[h$density > 0]
  bin_size = h$mids[2]-h$mids[1]
  entropy = -1*sum(log(p)*p*bin_size) 
  entropy_range = c(entropy_range, entropy)
}


plot(x= n_range, y= entropy_range, log = "x")
lines(x= c(1,10^6), y= c(1,1)*(0.5*log(8*pi)+0.5), lty = 2)

A tricky part is that the computation of entropy is different for continuous distributions

$$\int f(x) \log \left( f(x) \right) \text{d} x \approx \sum_{k = 1}^{n_{bins}} f_k \log \left( f_k \right) \cdot \text{bin_size}$$

When you just apply the bin probabilities $p_k = f_k \cdot \text{bin_size}$ and use the formula for discrete distributions then you get a different values.

$$\sum_{k = 1}^{n_{bins}} f_k \cdot \log \left( f_k \right) \cdot \text{bin_size} \neq \sum_{k = 1}^{n_{bins}} f_k \cdot \text{bin_size} \cdot \log \left( f_k \cdot \text{bin_size} \right) = \sum_{k = 1}^{n_{bins}} p_k \cdot \log \left( p_k \right)$$

You will be wrong with a factor $\log(\text{bin_size})$ and that explains the different curve displacements shown in your second edit.

In short: entropy is the expectation of the logarithm of the probability mass function or the probability density function $E[\log p(X)]$ and $E[\log f(X)]$. Problems arrise when we mix those formulas and functions.