0
$\begingroup$

For random variable $X$, entropy is calculated as $H(X) =-\sum_i p(x_i) \ln p(x_i)$. Differential entropy, not shown, is its continuous counterpart. Both use log probabilities which convert the original probabilities, whose range was $[0,1]$, to the log scale which changes the range to $[-\infty,0]$.

It not uncommon to see exponential entropy being used instead, $\exp( H(X))$, I think to circumvent a singularity at $H(X)=0$. As we know, taking the exponential of a log will undo the log.

Given that the raw probabilities have been upgraded to log probabilities in the entropy measure, what is the intuition behind further upgrading the entropy measure by undoing its logs using $\exp$?

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ $\exp(H)$ has an easy interpretation in terms of the (approximate) number of equally common categories. Suppose $k = 1, \dots, K$ categories are equally common; then $H = \sum^K_{k=1} p_k \ln (1/p_k) = \sum^K_{k=1} (1/K) \ln K = \ln K$ and exponentiating that gets you back to $K$. In some fields this is called a numbers equivalent. The interpretation is useful even if probabilities do vary somewhat. More broadly, whenever logarithms appear, it is never surprising that exponentials also have a role. $\endgroup$
    – Nick Cox
    Commented Aug 20, 2020 at 19:21
  • $\begingroup$ "Given that the raw probabilities have been upgraded to log probabilities" is not a helpful characterisation. I don't know what would be better for your purposes. $\endgroup$
    – Nick Cox
    Commented Aug 20, 2020 at 19:27
  • $\begingroup$ your example shows how exponential entropy is handy for the equally-weighted probabilities case. what does it do for lower entropy cases (not equally weighted)? $\endgroup$
    – develarist
    Commented Aug 20, 2020 at 19:32
  • 1
    $\begingroup$ @NickCox this seems to be an answer, isn't it? $\endgroup$
    – Tim
    Commented Aug 20, 2020 at 19:33
  • $\begingroup$ i'm not challenging the answer because I did vote it, since the best answers anyway are examples. just trying to explore the intuition more if there are more examples (answers) to throw at me/me at $\endgroup$
    – develarist
    Commented Aug 20, 2020 at 19:36

0

Reset to default

Browse other questions tagged or ask your own question.