What is the significance of the log base being 2 in entropy? What if we take e or 10 as the base?
$\begingroup$
$\endgroup$
1
asked Feb 21, 2018 at 4:26
-
1$\begingroup$ the question pops up regularly. the answer is that Shannon himself didn't see any significance in the log base, see stats.stackexchange.com/a/432305/36041 $\endgroup$– AksakalCommented Jan 14, 2021 at 18:20
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
Nothing much. Recall that we have $$\log_a b = \frac{\log_c b}{\log_c a}$$
Hence, if you use other base, such as $e$ and $10$, you can always convert to another base using a scalar multiplication.
Communication/ information was thought in terms of bits, hence the magical number, $2$.
answered Feb 21, 2018 at 6:26
-
2$\begingroup$ It kind of helps that digital systems are easily/commonly designed to manipulate binary inputs... It is not just history, practical applications matter too. Ternary computers never really took off for good reasons. $\endgroup$ Commented Feb 22, 2018 at 0:24
-
$\begingroup$
$\endgroup$
1
I'd like to cite that in the Elements of Information Theory by Covers:
If the base of the logarithm is b, we denote the entropy as $H_b(X)$.If the base of the logarithm is e, the entropy is measured in nats.Unless otherwise specified, we will take all logarithms to base 2, and hence all the entropies will be measured in bits.
And in lemma 2.1.2:
$H_b(X) = (log_b a)H_a(X)$ Proof:
$log_bp = log_ba\ log_ap$.The second property of entropy enables us to change the base of the logarithm in the definition. Entropy can be changed from one base to another by multiplying by the appropriate factor.
Hope it helps.
answered Feb 22, 2018 at 0:01
-
2$\begingroup$ +1 Incidentally, this is the same argument that C. E. Shannon makes in his foundational paper, "A Mathematical Theory of Communication". $\endgroup$– Sycorax ♦Commented Feb 22, 2018 at 0:34
$\begingroup$
$\endgroup$
Apart from the fact that using bits (letters or symbols that can have one of the pre-determined 2 values at a time) is the most convenient way of encoding data, there is no other significance. If there were no computers or IT, we might as well be using $\log_{10}$, and in that case, we would have needed much less number of letters (digits) for encoding. Or, a combination of all digits and Roman alphabets would have brought $\log_{36}$ into picture. And as stated in another answer, the conversion is straightforward.
But what is slightly more perplexing is the idea that our choice for the number of values the letters can have is not limited to natural numbers! In fact, using an $e$-valued letter is the most natural choice and such a letter is called a nat or natural unit of information. Again the conversion is straightforward. But if you wanted to go deeper, have a look at Information Theory - Rationale Behind Using Logarithm for Entropy, and Other Explanations.
