5
$\begingroup$

When reading papers on machine learning, I have found that authors would often reference the "Shannon entropy". Curiously, often times the equation given would be:

$$H(p) = -\sum\limits_{i = 1}^n p_i \ln(p_i)$$

For instance, see:

https://arxiv.org/pdf/1502.00326.pdf

https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-121.pdf

There are a lot more

The problem is that for anyone who has ever taken a course on information theory, the logarithmic term in the entropy definition is base $2$, not base $e$. So they are referring to some more like Gibbs entropy instead of Shannon entropy.

Whereas the definition in this paper is correct to me: http://www.fizyka.umk.pl/publications/kmk/08-Entropie.pdf

Has anyone else noticed this phenomenon? Would there be a problem if one used Gibbs entropy in place of Shannon's entropy?

$\endgroup$
  • 4
    $\begingroup$ It doesn't matter. Just units change from bits to nats. As long as you use consistently everywhere, you will not have problems. $\endgroup$ – Cagdas Ozgenc May 11 '17 at 18:41
  • 1
    $\begingroup$ Recall that $\log_a x = \frac{\log_b x}{\log_b a}$. $\endgroup$ – Matthew Gunn May 11 '17 at 19:08
6
$\begingroup$

It's not a problem. In fact Shannon himself suggested that other units could be used, see in his paper "A Mathematical Theory of Communication" the very first equation (bottom of page 1). Here's a quote from the paper:

In analytical work where integration and differentiation are involved the base e is sometimes useful. The resulting units of information will be called natural units. Change from the base a to base b merely requires multiplication by $\log_b a$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.