5
$\begingroup$

It is a rather common thing to be concerned with the variance of an estimator. For instance, confidence intervals for the mean can be constructed based on the standard error.

Often, however, we look at minimizing the variance of an estimator. This leads us to the minimum variance unbiased estimator, for example, or to the use of control variates or baselines (in reinforcement learning and optimal control). In machine learning, the variance of the stochastic gradient estimator is a major concern as well (especially when dealing with those from e.g. REINFORCE).

My question is why we look at the variance specifically, rather than the entropy, especially when minimization is concerned. I'd be interested in both practical and theoretical reasons. Ideally, we want the estimator to form a low entropy Dirac delta around the true value (or if the target is random, closely approximate it with zero density in areas away from the target's distribution). I feel that entropy is a more principled measure of "noisiness" and uncertainty than variance (for instance, an answer here has a few reasons for this).

I can think of two reasons why we are concerned with the variance rather than the entropy:

  1. Entropy is hard to estimate. This matters for practical rather than theoretical reasons, I suppose.

  2. Variance gives us uncertainty/dispersion that is (a) about the mean and (b) in the same units (once square rooted anyway) as the quantity itself. Since the mean is usually the estimator itself, the variance measures how spread out the data are from it. For (b), being in the same units lets us relate the uncertainty back to the original values. Nevertheless, it's not clear either of these matter for optimization.

Are there other reasons to use variance, or conversely, some interesting reasons why looking at entropy makes sense?

$\endgroup$
2
$\begingroup$

Here is an arXiv paper on the topic with many references. The paper assumes a Bayesian setting. The abstract reads:

The minimum error entropy (MEE) criterion has been successfully used in fields such as parameter estimation, system identification and the supervised machine learning. There is in general no explicit expression for the optimal MEE estimate unless some constraints on the conditional distribution are imposed. A recent paper has proved that if the conditional density is conditionally symmetric and unimodal (CSUM), then the optimal MEE estimate (with Shannon entropy) equals the conditional median. In this study, we extend this result to the generalized MEE estimation where the optimality criterion is the Renyi entropy or equivalently, the $\alpha$-order information potential (IP).

One of the referenced papers is this interesting looking (for me behind a paywall) with abstract:

A study of the use of entropy as a criterion function for analyzing the performance of sampled-data estimating systems is presented, and performance bounds are obtained for a broad class of such systems. The general result is that the difference between the entropy of the signal to be estimated and the entropy of its estimate based on the output of a noisy sensor can never be larger than the mutual information between the sensor input and output. An interesting, but not totally satisfactory, sufficient condition for attainment of the bound is developed.

$\endgroup$
  • 1
    $\begingroup$ Thanks! It seems Minimum Error Entropy Criterion is a good search criterion. $\endgroup$ – user3658307 Aug 15 at 2:12
1
$\begingroup$

As you write, entropy is very hard to estimate: estimating entropy reliably requires a larger number of sample than the variance. Furthermore the estimates are also subject to the so-called "sampling bias" which biases all entropy estimates downward in a distribution-dependent fashion. Several techniques are available for correcting this bias (you can find some references in this article), but they are non-trivial to understand and they further increase the variance of the estimates. In the case the variables are Gaussian (which is your assumption when computing the variance) the entropy reduces to a logarithmic function of the variance. However, if your distribution is not Gaussian, the variance might be of little use for quantifying the uncertainty of your distribution.

$\endgroup$
  • $\begingroup$ Thanks for your answer! Maybe add some references (e.g., for why debiasing entropy predictions necessarily increases variance of the estimators). And why do you say Gaussian is our assumption when computing variance (variance is defined for any RV)? Do you mean it is only a good uncertainty estimate in the Gaussian case? $\endgroup$ – user3658307 Aug 15 at 2:16
  • $\begingroup$ Also, for the sentence "if your distribution is not Gaussian, the variance might be of little use for quantifying the uncertainty of your distribution.", I agree (e.g. think of a GMM with two components moving apart - increasing variance but not entropy), but variance is still commonly used (e.g. for quantifying uncertainty in predictive distributions of large Bayesian ML models)! Any references on this as well? $\endgroup$ – user3658307 Aug 15 at 2:19
  • $\begingroup$ I have referenced an old paper of mine in the answer where you can find references to some bias correction techniques (only for binned estimators though). These references also explain why the correction increases the variance. Regarding the variance: it can be estimated for any probability, however, only in certain cases it provides meaningful information about the distribution, this is typically when the distribution is Gaussian or at least bell-shaped. Since this happens with many data and since it is so easy to compute, people often just use the variance and hope it works well... $\endgroup$ – Cesare Aug 15 at 6:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.