- In decision trees, I notice that there is a Gini value for each node of the tree. If we have p predictors, do we sum up each of the individual predictor's Gini index to arrive at the Gini index for the node? In the link above, we only calculate the value for each predictor, not for an entire node.
The Gini index is always computed on the TARGET variable, not the predictors, as it tells us how "pure" our node is based on the classes of the target variable. To know the Gini Index of a node, the predictors are irrelevant.
Predictors become relevant when we want to split, as we evaluate every possible split of every possible predictor, every time obtaining two child nodes, for which we compute the Gini (once again, only looking at the target variable).
By looking at the link you added, I can see how it can be confusing, as the author asked the gini index to be computed for some predictors instead than for the target class. That is something you can do of course (Gini is just a function you compute on categorical data), but it has no application in CART.
- Additionally, the above link implies that for quantitative variables, we still calculate a Gini index based on frequencies. Does this imply that for most quantitative predictors, the Gini index will be close to 1−1/n?
As before, I think you got confused from the link. Gini is an impurity index that is used for classification and it therefore cannot be applied to continuous variables, as one would do regression in those cases instead. In the example you give (from the link) however, one could interpret the integer values of the $a_3$ variable as classes, and use that variable as categorical. This would be an example of multiclass classification with very sparse data. Even though I think this has little impact in most actual problems, the answer is yes, in such sparse cases the node would be very impure, and Gini would approach $1-\frac{1}{n}$, which is its maximum.
- Also, when we split a node into two branches, we choose the split that results in the lowest Gini index of the split. The Gini index of the split is the weighted average of the two nodes' Gini index, weighted by number of observations in each node. Is my understanding correct?
Yes, that is correct. We take the split that gives us the biggest decrease in Gini, by subtracting the Gini of the original node and the weighted mean of the two children nodes.
- Finally, can someone explain to me the intuition behind why increasing homogeneity in the nodes allows us to classify something accurately? I'm guessing the intuition is that similar observations (based on their features) should lead to similar response values. And a tree helps us group similar observations? Is that the right reasoning
Again, what we are doing is increasing the homogeneity of the nodes based on the target variable!. Which means that we want (hypothetically) to have nodes that only contain elements of one target class. Indeed, Gini and Misclassification Error (which is 1-Accuracy) are maximised and minimized for the same points. In particular, Gini is 0 when we are not misclassifying any element, and it is maximum when we are misclassifying half of them (in a 2 class setting, or $\frac{n-1}{n}$ in an $n$ class setting).
The reason we use Gini instead of Misclassification Error during the splits is that Gini is strictly concave, and this allows us to have better splitting properties. In particular, If we split a node (imagine target classes are $[20,50]$) and the majority class stays the same in the children nodes (imagine the new nodes have $[20,30]$ and $[0,20]$) then the misclassification error stays the same, and we have no decrease even though our split is clearly making some progress (before and after the split we are still misclassifying $20$ elements out of the total $70$). Gini, being concave, takes this into account, and the Gini difference before and after a split is NEVER zero for non-trivial splits (i.e. it is only zero if the split is not actually separating anything).
Here's an image of the gini index curve and the orginal node of the example (red dot), the final nodes (red crosses), and the weighted average (green dot). We therefore have a decrease from the two dots.
Hope it helps!
