2
$\begingroup$

I've read many related articles and posts. The more I read, the more I got confused about 'Gini index' and 'Gini Impurity'. I understood the concept but it seems to me that these things are used differently by different people. ISLR book* (page 326) defines Gini Index as $\sum p_i(1– p_i)$ or $1 - \sum p_i^2$.

enter image description here

However, this (and many other articles) [the Same question has been asked in comments too by Shanu_not answered though] compute Gini by $ p^2+q^2$ formula for Binary classifier.

So, their Gini Impurity [ 1 $-$ Gini Index] is exactly the same as the Gini Index computed as per ISLR book.

Please let me know what am I missing. I realize that reading concepts after a long break is painful.

*Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani. (2013). An introduction to statistical learning : with applications in R. New York :Springer,

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Apart from looking at the formulas, the words purity and impurity are indicative (so long as they are used carefully). $\sum p_i^2$ is maximal (purity is highest) when there is just one category present and so the sum is the sum of $1^2$ and any number of $0^2$ and so just $1$. $1 - \sum p_i^2$ is minimal in the same case (impurity is lowest). $\endgroup$
    – Nick Cox
    Commented Jun 5, 2019 at 6:09
  • 1
    $\begingroup$ $p^2 + q^2$ is clearly not a general recipe, but applies only when there are two categories in play. $\endgroup$
    – Nick Cox
    Commented Jun 5, 2019 at 6:12
  • $\begingroup$ The Gini index formula is the $G$ you defined above. That $p^2 + q^2$ computes somehow purity, it is specific to two classes, and the $1$ from $G$ got removed because it is constant when you compare two nodes in a decision tree. Usually splitting criteria in decision trees use impurity measures: eg Gini index or entropy. An example here: stats.stackexchange.com/questions/44382/… $\endgroup$
    – Simone
    Commented Jun 5, 2019 at 6:29
  • $\begingroup$ ISLR won't be recognisable by all readers, so please give references in good academic style (authors, date, book title, publishers, place). $\endgroup$
    – Nick Cox
    Commented Jun 5, 2019 at 7:44
  • $\begingroup$ Thanks Nick, done that. $\endgroup$
    – Dr Nisha Arora
    Commented Jun 5, 2019 at 16:09

1 Answer 1

Reset to default
1
$\begingroup$

Usually, the terms Gini Index and Gini Impurity are used as synonyms. Indeed, when defined as $1-\sum p_i^2 $ it measures impurity - in the sense that it increases with impurity.

To me it looks like the link you gave uses an alternative, rather confusing definition, where they use Gini Index as a measure of purity, and Gini Impurity as 1-Index. This is something I had never seen in the literature, and it does not seem to recur anywhere else (I took a quick tour of links and definitions, and I could not find it anywhere else).

Therefore, I would rather use the definition you can find in Hastie/Tibshirani's book, as it is the most common. Indeed, we can trace that definition it back to Classification And Regression Trees (Breiman, 1984):

4.3.3 The Gini Criterion
[...]
In later work the Gini diversity index was adopted. This has the form: $$i(t) = \sum_{i\neq j}p(i|t)p(j|t)$$ and can also be written as $$i(t) = [...] =1-\sum_jp(j|t)^2$$

The original name is therefore Gini (diversity) index, but since it is a measure of impurity you may also call it Gini impurity.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for confirming. $\endgroup$
    – Dr Nisha Arora
    Commented Feb 3, 2020 at 5:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.