Log probabilities versus squared probabilities (entropy vs Gini)

Question

The advantage of log probabilities over direct probabilities, as discussed here and here, is that they make numerical values close to $0$ more easy to work with. (my question, instead of the links, focuses on why one measure, that doesn't use log probabilities, is widely approved in practice and preferable over a different measure, that does log them, despite the advantages of the latter)

The real advantage is in the arithmetic. Log probabilities are not as easy to understand as probabilities (for most people), but every time you multiply together two probabilities (other than 1×1=1), you will end up with a value closer to 0. Dealing with numbers very close to 0 can become unstable with finite precision approximations, so working with logs makes things much more stable and in some cases quicker and easier.

Basically log probabilities (which are used in Shannon entropy) are a work-around from naively multiplying probabilities together (as done with Gini measures).

Why then would Gini impurity (or Gini coefficient, which has a different formula) be preferable and more intuitive than Shannon entropy if it multiplies probabilities together?

• $\textit{Gini}: \mathit{Gini}(X) = 1 - \sum_{i=1}^{n}p(x)_i^2$
• $\textit{Entropy}: H(X) = -\sum_{i=1}^{n}p(x)_i\log p(x)_i$

Someone here said logarithms are too complicated to calculate, but I don't see how hard could it be, given that it is just a button on a calculator. And as said, log probabilities are more stable than multiplied/squared probabilities.

Note: the scope of my question is directed more towards non-classification problems dealing with the discretized histograms of continuous random variables, and real-valued numerical applications. but any explanation might be helpful

Answer 1

From the side of computational complexity

Your sentence:

Someone here said logarithms are too complicated to calculate, but I don't see how hard could it be, given that it is just a button on a calculator. And as said, log probabilities are more stable than multiplied/squared probabilities.

is not accurate in terms of implementation and pragmatism of obtained methods to compute the final results in algorithms on the computer. You can find the complexity of computing logarithm and multiplication in this source. As you can see, the complexity of computing logarithm for an $n$-digit number is $O(M(n) \log{n})$ (if using the arithmetic-geometric mean iteration method), which $M(n)$ is the complexity of multiplication method that has been used in the computing of the logarithm method. Hence, at least the logarithm method is slower than the multiplication method by a factor $\log{n}$. Notice that it can be exacerbated when using multiple times in a computation like as you have mentioned.