7
$\begingroup$

For random forest, we split the node by Gini impurity or entropy for a set of features. The RandomForestClassifier in sklearn, we can choose to split by using Gini or Entropy criterion. However, what I read about Extra-Trees Classifier, a random value is selected for the split (I guess then there is nothing to do with Gini or Entropy). The ExtraTreesClassifier from sklearn has the option to choose Gini or Entropy for the split. I am a little bit confused here.

$\endgroup$
8
$\begingroup$

One iteration of Random Forest:

  1. Select $m$ features randomly as a candidate set of splitting features
  2. Within each of these features, find "best" cutpoint, where "best" is defined by Gini / Entropy / whatever measure
  3. Now you have $m$ features paired with their optimal cutpoints. Choose as your splitting feature and cutpoint the pair that has the "best" performance with respect to Gini / Entropy / whatever measure

One iteration of Extremely Randomized Trees:

  1. Select $m$ features randomly as a candidate set of splitting features

  2. Within each of these features $F_i$, with $i \in {1, ...,m}$ draw a single random cutpoint uniformly from the interval $(min(F_i), max(F_i))$. Evaluate the performance of this feature with this cutpoint with respect to Gini / Entropy / whatever measure

  3. Now you have $m$ features paired with their randomly selected cutpoints. Choose as your splitting feature and cutpoint the pair that has the "best" performance with respect to Gini / Entropy / whatever measure
$\endgroup$
  • $\begingroup$ Amazing explanation, I've been struggling with this and this cleared it up completely. $\endgroup$ – Yu Chen May 20 at 17:21
  • $\begingroup$ I'm confused on on extra trees. I thought all extra trees did was draw a random subset of features, and then draw a random feature FROM that subset of features. I thought it didn't do any optimizing with respect to split metrics? $\endgroup$ – Michael Hsu Aug 8 at 17:40
  • $\begingroup$ It does optimize w/r/t split metrics, but only after those split metrics are randomly chosen. From scikit-learn's own documentation : "As in random forests, a random subset of candidate features is used, but instead of looking for the most discriminative thresholds, thresholds are drawn at random for each candidate feature and the best of these randomly-generated thresholds is picked as the splitting rule" $\endgroup$ – klumbard Aug 8 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.