When given samples of a discrete random variable, the entropy of the distribution may be estimated by $- \sum \hat{P_i} \log{\hat{P_i}}$, where $\hat{P_i}$ is the sample estimate of the frequency of the $i$th value. (this is up to a constant determined by the base of the log.) This estimate should not be applied to observations from a continuous distribution, at least naively, because it would yield a value which depends only on the sample size.

Beirlant et al describe a number of approaches for the continuous problem, including estimates based on empirical CDF, nearest neighbor distances and the $m$-spacing estimate, which is given by $$\frac{1}{n}\sum_{i=1}^{n-m}\log{(\frac{n}{m}(X_{(i+m)} - X_{(i)}))}$$, where $X_{(i)}$ is the $i$th order statistic of the sample, and $m$ varies in a certain way with $n$. It is not clear how this estimate is to be computed in the presence of ties, i.e. it does not appear to be applicable to discrete distributions. (a naive correction for ties (drop terms which have $\log{0}$) appears to give an estimator which does not depend on the relative frequency of the classes, only their values, which seems wrong.)

The question: is there an 'all-purpose' estimator which can deal with both discrete and continuous (or even mixed) distributions?

  • $\begingroup$ I'm not immediately sure how to solve this, but it seems reasonable that an estimate of the cdf can still be obtained. It might not be continuous, but I don't see that as a huge problem. $\endgroup$ Sep 15, 2010 at 0:11
  • $\begingroup$ Like Robin says, definition of entropy depends on choice of measure. For discrete distributions it's clear-cut, but for continuous there's no agreement on the right measure to use, so you have a spectrum of different entropies. Perhaps you should ask why you need entropy in a first place, ie, what exactly do you expect to learn from it? $\endgroup$ Sep 15, 2010 at 17:06
  • $\begingroup$ There are some significant differences between entropy of a discrete measure and differential entropy, as calculated for a measure with a density function. Estimating entropy is actually a difficult problem with lots of pretty strong negative results. See the work of Liam Paninski, for example. Researchers in neuroscience seem to take a strong interest in problems of this type. $\endgroup$
    – cardinal
    Aug 3, 2011 at 12:16
  • $\begingroup$ One particular paper you might interested in is: L. Paninski, Estimation of entropy and mutual information, Neural Computation, vol. 15, 1191-1254. $\endgroup$
    – cardinal
    Aug 3, 2011 at 12:21
  • $\begingroup$ @yaroslav - even for discrete distributions, definition of which counting measure to use is not clear cut. For example take an urn with $N$ balls, $W$ white and $B=N-W$ black. We can count "outcomes", which gives $Pr(W=w|N)=\frac{1}{N+1}$, or alternatively we can count individual balls, which gives $Pr(W=w|N)=2^{-N}{N \choose w}$ $\endgroup$ Oct 9, 2011 at 4:24

1 Answer 1


Entropy is entropy with respect to a measure

As noticed in the answer to this question https://mathoverflow.net/questions/33088/entropy-of-a-general-prob-measure/33090#33090 , entropy is only defined with respect to a given measure. For example discrete entropy is entropy with respect to counting measure.

Sample entropy should be an estimate of a predefined entropy.

I think the idea of sample entropy can be generalised to any type of entropy but you need to know which entropy you are trying to estimate before estimating it.

Example of entropy with respect to counting+lebesgues

For example, if you are trying to estimate the entropy with respect to the sum of the lebesgues measure on $[0,1]$ and the counting measure on $\{0,1\}$ a good estimate certainly (my intuition) is a sum of the two estimates you mention in your question (with $i=0,1$ in the first sum).

  • $\begingroup$ very good. I'll sum the two and see how that works! $\endgroup$
    – shabbychef
    Sep 16, 2010 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.