Why consider the variance rather than the entropy of estimators?

Question

It is a rather common thing to be concerned with the variance of an estimator. For instance, confidence intervals for the mean can be constructed based on the standard error.

Often, however, we look at minimizing the variance of an estimator. This leads us to the minimum variance unbiased estimator, for example, or to the use of control variates or baselines (in reinforcement learning and optimal control). In machine learning, the variance of the stochastic gradient estimator is a major concern as well (especially when dealing with those from e.g. REINFORCE).

My question is why we look at the variance specifically, rather than the entropy, especially when minimization is concerned. I'd be interested in both practical and theoretical reasons. Ideally, we want the estimator to form a low entropy Dirac delta around the true value (or if the target is random, closely approximate it with zero density in areas away from the target's distribution). I feel that entropy is a more principled measure of "noisiness" and uncertainty than variance (for instance, an answer here has a few reasons for this).

I can think of two reasons why we are concerned with the variance rather than the entropy:

1. Entropy is hard to estimate. This matters for practical rather than theoretical reasons, I suppose.

2. Variance gives us uncertainty/dispersion that is (a) about the mean and (b) in the same units (once square rooted anyway) as the quantity itself. Since the mean is usually the estimator itself, the variance measures how spread out the data are from it. For (b), being in the same units lets us relate the uncertainty back to the original values. Nevertheless, it's not clear either of these matter for optimization.

Are there other reasons to use variance, or conversely, some interesting reasons why looking at entropy makes sense?

Answer 1

Here is an arXiv paper on the topic with many references. The paper assumes a Bayesian setting. The abstract reads:

The minimum error entropy (MEE) criterion has been successfully used in fields such as parameter estimation, system identification and the supervised machine learning. There is in general no explicit expression for the optimal MEE estimate unless some constraints on the conditional distribution are imposed. A recent paper has proved that if the conditional density is conditionally symmetric and unimodal (CSUM), then the optimal MEE estimate (with Shannon entropy) equals the conditional median. In this study, we extend this result to the generalized MEE estimation where the optimality criterion is the Renyi entropy or equivalently, the $\alpha$-order information potential (IP).

One of the referenced papers is this interesting looking (for me behind a paywall) with abstract:

A study of the use of entropy as a criterion function for analyzing the performance of sampled-data estimating systems is presented, and performance bounds are obtained for a broad class of such systems. The general result is that the difference between the entropy of the signal to be estimated and the entropy of its estimate based on the output of a noisy sensor can never be larger than the mutual information between the sensor input and output. An interesting, but not totally satisfactory, sufficient condition for attainment of the bound is developed.

Answer 2

As you write, entropy is very hard to estimate: estimating entropy reliably requires a larger number of sample than the variance. Furthermore the estimates are also subject to the so-called "sampling bias" which biases all entropy estimates downward in a distribution-dependent fashion. Several techniques are available for correcting this bias (you can find some references in this article), but they are non-trivial to understand and they further increase the variance of the estimates. In the case the variables are Gaussian (which is your assumption when computing the variance) the entropy reduces to a logarithmic function of the variance. However, if your distribution is not Gaussian, the variance might be of little use for quantifying the uncertainty of your distribution.