I'm trying to understand the theory behind decision trees (CART) and especially the scikit-learn implementation.
CART constructs binary trees using the feature and threshold that yield the largest information gain at each node. scikit-learn documentation
Furthermore it defines Gini Impurity and Entropy Impurity as follows:
And that I should
select the parameters that minimises the impurity.
However in the specific
DecisionTreeClassifier I can choose the
Supported criteria are “gini” for the Gini impurity and “entropy” for the information gain. DecisionTreeClassifier
What I don't understand is that (in my opinion) information gain is the difference of the impurity of the parent node and the weighted average of the left and right childs. This means that I should choose the feature with the highest gain for the split.
Is information gain only applicable for entropy or also for gini impurity? According to the classifier criterion its either gini impurity or entropy information gain which would mean that it either minimizes gini impurity or maximizes information gain?