How to derive the bias of an entropy estimate I am trying to understand the bias-variance trade-off in the context of non-parametric entropy estimation.
Specifically using a histogram approach to estimate the entropy of a sample we have:
$$\hat{H} = - \sum^{B}_{i=1}\hat{p}_iv_i\log(\hat{p}_i) $$
(for a generic partition of $B$ bins where $\hat{p}_i$ is the estimate of probability density in bin $i$ and $v_i$ is the volume of the bin $i$).
I understand that bias is defined as: $E[\hat{H}] - H$, where $H$ is the 'true' entropy, but in the general case one doesn't know the true population distribution (hence the nonparametric estimation). Hence I don't understand how bias is calculated in general for this estimator, nor how it changes with the parameter $B$?
Secondly, the variance is given by: $E[E[\hat{H}] - \hat{H}]$, which avoids the above problem but how does the expectation value $E[\hat{H}]$ even differ from the estimator $\hat{H}$?
Finally does it even make sense to consider variance outside the context of 'training' the histogram estimator - if you assume there is just one sample and one is trying to get a close estimate to the true value, it doesn't feel like over-fitting is a concern and one should aim for the minimum-bias parameterisation.
I think I am missing some really basic context here as these simple concepts are not making sense to me.
 A: First, don't forget that the bias-variance tradeoff is in terms of the expected values in application of your model to new samples from the underlying population. See this answer. You presumably want a model to work well on the population, not just n your data set.
Second, the form of the downward bias of the plug-in estimator of the Shannon entropy that you show has been known since the 1950s. See this answer and its links for details.
Third, the bootstrap can provide non-parametric estimates of the bias; that was one of the reasons for developing the bootstrap. Under the bootstrap principle, taking multiple bootstrap samples of the original size (with replacement) from your data set mimics the process of taking your original sample from the underlying population. So the difference between the entropy estimates on the bootstrapped samples and the entropy estimate on the original sample gives an estimate of the bias of your original entropy estimate. See this answer for more details and links. In your application, you could try different bin sizes/numbers and evaluate how that bias changes with the number of bins $B$, given the total size of your data set.
