For random forest, we split the node by Gini impurity or entropy for a set of features. The RandomForestClassifier in sklearn, we can choose to split by using Gini or Entropy criterion. However, what I read about Extra-Trees Classifier, a random value is selected for the split (I guess then there is nothing to do with Gini or Entropy). The ExtraTreesClassifier from sklearn has the option to choose Gini or Entropy for the split. I am a little bit confused here.
1 Answer
$\begingroup$
$\endgroup$
3
One iteration of Random Forest:
- Select $m$ features randomly as a candidate set of splitting features
- Within each of these features, find "best" cutpoint, where "best" is defined by Gini / Entropy / whatever measure
- Now you have $m$ features paired with their optimal cutpoints. Choose as your splitting feature and cutpoint the pair that has the "best" performance with respect to Gini / Entropy / whatever measure
One iteration of Extremely Randomized Trees:
Select $m$ features randomly as a candidate set of splitting features
Within each of these features $F_i$, with $i \in {1, ...,m}$ draw a single random cutpoint uniformly from the interval $(min(F_i), max(F_i))$. Evaluate the performance of this feature with this cutpoint with respect to Gini / Entropy / whatever measure
- Now you have $m$ features paired with their randomly selected cutpoints. Choose as your splitting feature and cutpoint the pair that has the "best" performance with respect to Gini / Entropy / whatever measure
-
$\begingroup$ Amazing explanation, I've been struggling with this and this cleared it up completely. $\endgroup$– Yu ChenCommented May 20, 2019 at 17:21
-
$\begingroup$ I'm confused on on extra trees. I thought all extra trees did was draw a random subset of features, and then draw a random feature FROM that subset of features. I thought it didn't do any optimizing with respect to split metrics? $\endgroup$ Commented Aug 8, 2019 at 17:40
-
$\begingroup$ It does optimize w/r/t split metrics, but only after those split metrics are randomly chosen. From scikit-learn's own documentation : "As in random forests, a random subset of candidate features is used, but instead of looking for the most discriminative thresholds, thresholds are drawn at random for each candidate feature and the best of these randomly-generated thresholds is picked as the splitting rule" $\endgroup$– klumbardCommented Aug 8, 2019 at 18:33