If we want to calculate the shannon entropy of a text, considering that the elementary symbols aren't characters but words, will we still use a base-2 logarithm ? Or will we use a base-n logarithm where n is the number of words present in the vocabulary set ?


1 Answer 1


It doesn't matter which base you use, as long as you're consistent about it. This is true whether you’re modeling characters, words, or any other discrete set of symbols.

You switch from one base to another by multiplying by a constant. (See the change of base formula.) That means that switching from one base to another is a change of units:

  • Base 2 measures entropy in bits.
  • Base $e$ (the natural logarithm) measures entropy in nats.
  • Base 10 measures entropy in hartleys (though this is rarely used).

Just as it doesn't make sense to directly compare $x$ feet and $y$ meters, you shouldn't compare $x$ bits and $y$ nats; instead you should convert them to the same units.

Why do you think it would need to be base-$n$? I've never seen a resource that suggests this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.