If we want to calculate the shannon entropy of a text, considering that the elementary symbols aren't characters but words, will we still use a base-2 logarithm ? Or will we use a base-n logarithm where n is the number of words present in the vocabulary set ?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It doesn't matter which base you use, as long as you're consistent about it. This is true whether you’re modeling characters, words, or any other discrete set of symbols.
You switch from one base to another by multiplying by a constant. (See the change of base formula.) That means that switching from one base to another is a change of units:
- Base 2 measures entropy in bits.
- Base $e$ (the natural logarithm) measures entropy in nats.
- Base 10 measures entropy in hartleys (though this is rarely used).
Just as it doesn't make sense to directly compare $x$ feet and $y$ meters, you shouldn't compare $x$ bits and $y$ nats; instead you should convert them to the same units.
Why do you think it would need to be base-$n$? I've never seen a resource that suggests this.
answered Jun 11, 2021 at 22:41