2
$\begingroup$

The advantage of log probabilities over direct probabilities, as discussed here and here, is that they make numerical values close to $0$ more easy to work with. (my question, instead of the links, focuses on why one measure, that doesn't use log probabilities, is widely approved in practice and preferable over a different measure, that does log them, despite the advantages of the latter)

The real advantage is in the arithmetic. Log probabilities are not as easy to understand as probabilities (for most people), but every time you multiply together two probabilities (other than 1×1=1), you will end up with a value closer to 0. Dealing with numbers very close to 0 can become unstable with finite precision approximations, so working with logs makes things much more stable and in some cases quicker and easier.

Basically log probabilities (which are used in Shannon entropy) are a work-around from naively multiplying probabilities together (as done with Gini measures).

Why then would Gini impurity (or Gini coefficient, which has a different formula) be preferable and more intuitive than Shannon entropy if it multiplies probabilities together?

  • $\textit{Gini}: \mathit{Gini}(X) = 1 - \sum_{i=1}^{n}p(x)_i^2$
  • $\textit{Entropy}: H(X) = -\sum_{i=1}^{n}p(x)_i\log p(x)_i$

Someone here said logarithms are too complicated to calculate, but I don't see how hard could it be, given that it is just a button on a calculator. And as said, log probabilities are more stable than multiplied/squared probabilities.

Note: the scope of my question is directed more towards non-classification problems dealing with the discretized histograms of continuous random variables, and real-valued numerical applications. but any explanation might be helpful

$\endgroup$
1
  • 2
    $\begingroup$ Please give sources for your quotations. Those who don't care can ignore the sources, but others will be helped by giving sources, say to follow up an interesting quotation or even to check on context or form a view of whether you are quoting a competent authority. $\endgroup$
    – Nick Cox
    Oct 22 '20 at 8:19
2
$\begingroup$

From the side of computational complexity

Your sentence:

Someone here said logarithms are too complicated to calculate, but I don't see how hard could it be, given that it is just a button on a calculator. And as said, log probabilities are more stable than multiplied/squared probabilities.

is not accurate in terms of implementation and pragmatism of obtained methods to compute the final results in algorithms on the computer. You can find the complexity of computing logarithm and multiplication in this source. As you can see, the complexity of computing logarithm for an $n$-digit number is $O(M(n) \log{n})$ (if using the arithmetic-geometric mean iteration method), which $M(n)$ is the complexity of multiplication method that has been used in the computing of the logarithm method. Hence, at least the logarithm method is slower than the multiplication method by a factor $\log{n}$. Notice that it can be exacerbated when using multiple times in a computation like as you have mentioned.

$\endgroup$
18
  • 1
    $\begingroup$ @develarist my pleasure. Please search "Arithmetic–geometric mean iteration" in the source of the link. $\endgroup$
    – OmG
    Oct 21 '20 at 14:02
  • 2
    $\begingroup$ I find this answer irrelevant to most statistical applications because (a) the asymptotic complexity of computing logarithms doesn't matter and (b) often those logarithms are obtained through initial data (such as an algorithm that directly returns the log of a probability) and from then on, only addition of logarithms is required (as in computing a log likelihood). $\endgroup$
    – whuber
    Oct 21 '20 at 14:31
  • 1
    $\begingroup$ @develarist Gini coefficient was designed a century ago as a measure of heterogeneity. depending on what you use it for, applying it to binned numerical variables may or may give the same results as entropy, they are quite consistent metrics $\endgroup$
    – carlo
    Oct 21 '20 at 17:17
  • 1
    $\begingroup$ @Carlo Please explain how the number of digits in the computer's representation of numbers is relevant to the performance of random forests! $\endgroup$
    – whuber
    Oct 21 '20 at 17:31
  • 1
    $\begingroup$ @Carlo As I explained in an earlier comment, it is rare that computing logarithms is even an issue: in many, if not most, statistical applications the logarithms are already available and one is only taking linear combinations of them. Perhaps we are agreeing with each other, though, as your comment about the asymptotics suggests. $\endgroup$
    – whuber
    Oct 21 '20 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.