A simple & clear explanation of the Gini impurity? In a context of decision tree splitting, it is not obvious to see why the Gini impurity
$$
i(t)=1-\sum\limits_{j=1}^k p^2(j|t)
$$
is a measure of node t impurity. Is there an easy explanation of this?
 A: Imagine an experiment with $k$ possible output categories. Category $j$ has a probability of occurrence $p(j|t)$ (where $j=1,..k$)
Reproduce the experiment two times and make these observations:

*

*the probability of obtaining two identical outputs of category $j$ is $$ p^2(j|t) $$

*the probability of obtaining two identical outputs, independently of their category, is: $$\sum\limits_{j=1}^k p^2(j|t)$$

*the probability of obtaining two different outputs is thus: $$1-\sum\limits_{j=1}^k p^2(j|t)$$
That's it: the Gini impurity is simply the probability of obtaining two different outputs, which is an "impurity measure".

Remark: another expression of the Gini index is:
$$
\sum\limits_{j=1}^k p_j(1-p_j)
$$
This is the same quantity:
$$
\sum\limits_{j=1}^k p_j(1-p_j) = \left(\sum\limits_{j=1}^k p_j \right) -\left( \sum\limits_{j=1}^k p^2_j \right) = 1 - \sum\limits_{j=1}^k p^2_j 
$$
A: Gini impurity = logical entropy = Gini-Simpson biodiversity index = quadratic entropy with logical distance function (1-Kroneckerdelta), etc.
See: Ellerman, David. 2018. “Logical Entropy: Introduction to Classical and Quantum Logical Information Theory.” Entropy 20 (9): Article ID 679. https://doi.org/10.3390/e20090679, and the references contained therein.
