A simple & clear explanation of the Gini impurity?

Question

In a context of decision tree splitting, it is not obvious to see why the Gini impurity $$ i(t)=1-\sum\limits_{j=1}^k p^2(j|t) $$ is a measure of node t impurity. Is there an easy explanation of this?

Answer 1

Imagine an experiment with $k$ possible output categories. Category $j$ has a probability of occurrence $p(j|t)$ (where $j=1,..k$)

Reproduce the experiment two times and make these observations:

• the probability of obtaining two identical outputs of category $j$ is $$ p^2(j|t) $$
• the probability of obtaining two identical outputs, independently of their category, is: $$\sum\limits_{j=1}^k p^2(j|t)$$
• the probability of obtaining two different outputs is thus: $$1-\sum\limits_{j=1}^k p^2(j|t)$$

That's it: the Gini impurity is simply the probability of obtaining two different outputs, which is an "impurity measure".

---

Remark: another expression of the Gini index is: $$ \sum\limits_{j=1}^k p_j(1-p_j) $$ This is the same quantity: $$ \sum\limits_{j=1}^k p_j(1-p_j) = \left(\sum\limits_{j=1}^k p_j \right) -\left( \sum\limits_{j=1}^k p^2_j \right) = 1 - \sum\limits_{j=1}^k p^2_j $$

Answer 2

Gini impurity = logical entropy = Gini-Simpson biodiversity index = quadratic entropy with logical distance function (1-Kroneckerdelta), etc. See: Ellerman, David. 2018. “Logical Entropy: Introduction to Classical and Quantum Logical Information Theory.” Entropy 20 (9): Article ID 679. https://doi.org/10.3390/e20090679, and the references contained therein.