1
$\begingroup$

Entropy is the measure of randomness of a random variable. $$H(X) = -\sum_{x\in X}p(x)\log_{2}(p(x))$$

The units when using the $\log_{2}$ is bits i.e., how many bits required to store the information present in the random variable $X$.

What will be the units of entropy when use other base for log i.e., natural log/log base 10 or any other? How we will interpret those units?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

For natural log the units are called "nats". I believe it's just a convention to define entropy with natural log and it probably stems from thermodynamic entropy which uses nats for convenience: as wiki puts it "Physical systems of natural units that normalize Boltzmann's constant to 1 are effectively measuring thermodynamic entropy in nats".

As the main concern about entropy is its role in definition of mutual information between random variables, there's no practical effect of using different bases for log.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ +1 Nice response. My question may seem like a bit of a tangent but here it is. In a recent paper by Lin and Tegmark, Critical Behavior from Deep Dynamics, available here ... ai2-s2-pdfs.s3.amazonaws.com/5ba0/… , they discuss the consequences for NLP models in positing distributional assumptions involving "nat" processes and, specifically, what this can mean for truly power-lawed processes and information-theoretic metrics such as Kullback-Leibler divergence. Extending that to entropy, do you think their concerns are relevant-is the "nat" right? $\endgroup$
    – user78229
    Commented Dec 25, 2016 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.