6
$\begingroup$

Differential entropy (the continuous version of Shannon's entropy measure) is

$$ H = - \int_{-\infty}^\infty f(x) \log f(x) \mathrm{d}x, $$ where $f(x)$ is a probability density function.

What is the intuition behind computing the exponential entropy of this? Are the properties of the original improved?

$$ \exp(H) = \exp\Bigg[ -\int_{-\infty}^\infty f(x) \log f(x) \mathrm{d}x \Bigg] $$

I'm guessing that the exponentiation means something, but what?


According to Cover and Thomas (1991), entropy as a measure of uncertainty is:

  • homogeneous
  • not left bounded
  • not sub-additive

therefore, it lacks three of four desirable properties of coherent risk measures. The exponential function attempts to account for these issues but does not achieve this adequately.

$\endgroup$
8
  • 1
    $\begingroup$ In Boltzmann's definition of entropy, it's proportional to the log of the number of possible micro-states leading to the same macro-state. So... maybe exp of entropy means something analogous to the number of micro-states leading to the same macro-state. see (en.wikipedia.org/wiki/…) $\endgroup$ Aug 3 '20 at 8:37
  • $\begingroup$ when taken out of its native thermodynamics context, how else can i think of these micro and macro states? i.e. any applications of Boltzmann's principle outside of physics $\endgroup$
    – develarist
    Aug 3 '20 at 8:48
  • $\begingroup$ I'll elaborate in an answer, but it's just my two cents $\endgroup$ Aug 3 '20 at 9:00
  • 3
    $\begingroup$ This expression is standard in mathematics, although a bit obscure: relative to the distribution given by $f,$ it is the geometric mean of the density $f.$ Generally, the geometric mean of a positive variable $X$ is $\exp(E[\log(X)]).$ $\endgroup$
    – whuber
    Aug 3 '20 at 13:17
  • 3
    $\begingroup$ @Itamar Most people would misunderstand what you mean by "values of the distribution," I suspect. The exponential of the (differential) entropy is the geometric mean of the distribution densities. The discrete analog is the geometric mean of the distribution probabilities. The associated values are not involved. $\endgroup$
    – whuber
    Aug 3 '20 at 15:13
7
$\begingroup$

I will begin with building intuitions for the discrete case and then discuss the continuous case.

The discrete case

First, consider exponential entropy for the special case of a discrete uniform distribution $U^N$ over $N$ outcomes, i.e. $U^N_i = \frac{1}{N}$. It's easy to show that exponential entropy is equal to the number of outcomes $N$: \begin{align} \exp\left(H\left(U^N\right)\right)& = \exp\left(-\sum_i U^N_i \ln(U^N_i)\right)\\ & = \exp\left(-\sum_i \frac{1}{N} \ln\left(\frac{1}{N}\right)\right)\\ & = \exp\left(N \frac{1}{N} \ln\left(N\right)\right)\\ & = N \end{align} For an arbitrary probability distribution over $M$ outcomes $P^M$, there is then some number $N \leq M$ such that: \begin{align} N = \exp\left(H\left(U^N\right)\right) \leq \exp\left(H\left(P^M\right)\right) \leq \exp\left(H\left(U^{N+1}\right)\right) = N + 1 \end{align} where equal $N = M$ just in case $P^M$ is uniform.

From this inequality, we can interpret exponential entropy as the effective number of outcomes: The probability distribution $P^M$ has about as much uncertainty as a uniform distribution over $\left\lfloor\exp\left(H\left(P^M\right)\right)\right\rfloor$ or $\left\lceil\exp\left(H\left(P^M\right)\right)\right\rceil$ outcomes. Intuitively, a probability distribution with exponential entropy near 2 is about as uncertain as a fair coin flip, and a probability distribution with exponential entropy near one is nearly deterministic.

Exponential entropy is sometimes called perplexity. In this context, the base of the exponent and logarithm are typically written as 2 rather than $e$, but it doesn't matter since $2^{\log_2(x)} = e^{\log_e(x)} = x$.

Predicting a sample

We can use these metrics and intuitions for understanding how well a probability distribution predicts a sample. Call the true data distribution $P$, and the distribution we are measuring $Q$. In a typical use case, $Q$ is a model we have estimated, and now we want to measure how well it fits data that is distributed according to $P$. The cross-entropy of $Q$ relative to $P$ is: \begin{align} H(P, Q) & = -\sum_i P_i \ln Q_i \end{align} In this typical use case, we cannot compute the cross-entropy exactly because we do not know $P$ (otherwise we would use $P$ instead of estimating $Q$). Instead, we gather a dataset $D$, or sample, that is distributed according to $P$, and perform a Monte-carlo estimate of $H(P, Q)$ by averaging across the dataset: \begin{align} H(P, Q) & = -\sum_i P_i \ln Q_i \\ & \approx -\frac{1}{T} \sum_{i\sim P_i} \ln Q_i \\ & = -\frac{1}{T} \sum_{i\in D} \ln Q_i \end{align} where $D$ is just a dataset containing $T$ observations that we are treating as a random sample from the true distribution (Note that $D$ may contain duplicate entries, and may lack some entries entirely).

Note that $H(P, Q) \geq H(P)$, with equality just in case $P=Q$, so lower cross-entropy indicates that $Q$ is closer to $P$. If we exponentiate the cross-entropy to get the perplexity, we see how uncertain the distribution was on average when predicting each observation. A typical application is language modeling: if the perplexity is 100, then, on average, the model was as uncertain in predicting the next word as if it were choosing uniformly among 100 possible next words.

Note that $D$ can be a different sample (still from $P$) from the one that used used to estimate $Q$. In this case, the perplexity is held-out and provides a measure of how well the model generalizes to unseen data from the same distribution it was estimated on, and can be compared to the perplexity on the estimation dataset to assess whether your model has overfit the estimation data.

The continuous case

Shannon obtained the continuous version of entropy in your post by simply replacing the summation sign with an integral rather than performing a rigorous derivation. You can approximate a continuous distribution by binning the random variable and then defining a probability distribution over the bins, with the approximation improving as the number of bins increases. In this sense, you can view the exponential entropy of the approximating distribution in a similar way.

Unfortunately, as the number of bins goes to infinity to make the discrete distribution approach the continuous distribution in the limit, you end up with an inconvenient infinity in the expression. On reflection, this is not so surprising, as the probability of a single real number under a continuous distribution is zero.

$\endgroup$
4
  • $\begingroup$ perplexity measures how well a probability distribution predicts the sample. how can this be explained differently and what is the intuition, because I only know about sampling from probability distributions, not using them to predict samples? After all, I am only measuring a distribution based on given data, call it sample #1, there is no need for me to further predict another sample #2 $\endgroup$
    – develarist
    Aug 21 '20 at 9:36
  • $\begingroup$ I've added a sub-section that addresses prediction accuracy. Is it clear? $\endgroup$
    – jkpate
    Aug 21 '20 at 18:19
  • $\begingroup$ so perplexity is used to evaluate how well a probability distribution, what you call $P$, which is proxied by available data $D$, predicts some sample, $Q$, with $Q$ also being the estimated model. In summary, how an empirical distribution predicts a model. How can I make sense of this if I am used to models being the ones to make predictions? don't you mean $Q$ are the model predictions, and $P$ is the model that you would like to fit on those predictions? $\endgroup$
    – develarist
    Aug 24 '20 at 6:17
  • $\begingroup$ A perplexity example that uses exponential entropy rather than cross-entropy would be nice. but given that perplexity is all about predicting a sample, a second object, as what the cross-entropy example demonstrates, it seems like perplexity in fact applies only to measures that use two objects as inputs, such as cross-entropy and KL divergence? whereas exponential entropy only uses a single object input $\endgroup$
    – develarist
    Aug 24 '20 at 6:19
2
$\begingroup$

Exponential entropy measures the extent of a distribution, and can be used to avoid the case of singularity when the weighted average entropy of some variables is zero, $\bar{H}(X) = 0$.

Campbell, L. “Exponential Entropy as a Measure of Extent of a Distribution.” Z. Wahrscheinlichkeitstheorie verw., 5 (1966), pp. 217–225.

$\endgroup$
3
  • 1
    $\begingroup$ What does extent mean here? $\endgroup$ Aug 21 '20 at 10:58
  • $\begingroup$ not sure but it is described in the article and meant to be different than the spread of a distribution $\endgroup$
    – develarist
    Aug 21 '20 at 11:02
  • 1
    $\begingroup$ Thanks. I just thought, if the OP is interested in what exponential entropy is and the intuition behind, leaving the concept of extent unexplained does not help much. But at least there is the reference... $\endgroup$ Aug 21 '20 at 11:22
2
$\begingroup$

It's just my two cents, but I can think of an interpretation, following part of the development of the KL divergence and working from it:

Let's consider the discrete case, with a probability distribution $p_1...p_n$.
Its entropy is $S = -\sum _i p_i \log p_i$ (just the discrete form of what you posted).

Now, let's say we have $N$ variables following this distribution. The probability for $m_1$ of them to have value $1$, $m_2$ to have value $2$ and so forth is $ H= \prod_i {p_i}^{m_i} $ (where $\sum_i m_i =N$).

Now, if we ask what's the probability of those $m$'s to have the same proportions as the probability distribution (i.e. $m_i = Np_i$; never mind m being an integer), we have $H=\prod_i {p_i}^{N p_i} =(\prod_i {p_i}^{p_i})^N $

We can define the inner expression as $H_1$, having $H = H_1 ^N $; you can see that $-\log H_1 = S$.
This allows us to understand the exponent of the entropy as the (inverse of the) probability of a sample drawn from a distribution to follow the same proportion as that distribution (properly corrected for sample size).

$\endgroup$
2
  • $\begingroup$ Where is the correspondence with micro and macro states mentioned? The continuous case might make more sense to me $\endgroup$
    – develarist
    Aug 3 '20 at 15:43
  • $\begingroup$ It's not mentioned here explicitly, but note that "variable $1$ has value $v_1$, variable $2$ has value $v_2$... etc" is a micro-state, and this derivation deals directly with macro-states ("$m_1$ variables have value $1$... etc") $\endgroup$ Aug 4 '20 at 8:35
2
$\begingroup$

Entropy can be used as a measure of diversity, as biodiversity in ecology, or of income inequality, ... see for instance How is the Herfindahl-Hirschman index different from entropy?. In ecology one is then interested in the effective number of species, and it turns out this is given as the exponential of entropy, see How to include the observed values, not just their probabilities, in information entropy?.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.