# Is “Shannon entropy” used incorrectly in machine learning related literature?

When reading papers on machine learning, I have found that authors would often reference the "Shannon entropy". Curiously, often times the equation given would be:

$$H(p) = -\sum\limits_{i = 1}^n p_i \ln(p_i)$$

For instance, see:

https://arxiv.org/pdf/1502.00326.pdf

https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-121.pdf

There are a lot more

The problem is that for anyone who has ever taken a course on information theory, the logarithmic term in the entropy definition is base $2$, not base $e$. So they are referring to some more like Gibbs entropy instead of Shannon entropy.

Whereas the definition in this paper is correct to me: http://www.fizyka.umk.pl/publications/kmk/08-Entropie.pdf

Has anyone else noticed this phenomenon? Would there be a problem if one used Gibbs entropy in place of Shannon's entropy?

• It doesn't matter. Just units change from bits to nats. As long as you use consistently everywhere, you will not have problems. – Cagdas Ozgenc May 11 '17 at 18:41
• Recall that $\log_a x = \frac{\log_b x}{\log_b a}$. – Matthew Gunn May 11 '17 at 19:08

In analytical work where integration and differentiation are involved the base e is sometimes useful. The resulting units of information will be called natural units. Change from the base a to base b merely requires multiplication by $\log_b a$.