ExtraTreesClassifier is like a brother of RandomForest but with 2 important differences.
We are building multiple decision trees. For building multiple trees, we need multiple datasets. Best practice is that we don't train the decision trees on the complete dataset but we train only on fraction of data (around 80%) for each tree. In a random forest, we draw observations with replacement. So we can have repetition of observations in a random forest. In an ExtraTreesClassifier, we are drawing observations without replacement, so we will not have repetition of observations like in random forest.
The split is the process of converting a non-homogeneous parent node into 2 homogeneous child node (best possible). In RandomForest, it select the best split to convert the parent into the two most homogeneous child nodes. In an ExtraTreesClassifier, it selects a random split to divide the parent node into two random child nodes.
Let’s look at some ensemble methods ordered from high to low variance, ending in ExtraTreesClassifier.
1. Decision Tree (High Variance)
A single decision tree is usually overfits the data it is learning from because it learn from only one pathway of decisions. Predictions from a single decision tree usually don’t make accurate predictions on new data.
2. Random Forest (Medium Variance)
Random forest models reduce the risk of overfitting by introducing randomness by:
- building multiple trees (n_estimators)
- drawing observations with replacement (i.e., a bootstrapped sample)
- splitting nodes on the best split among a random subset of the features selected at every node. Split is process to convert non-homogeneous parent node into 2 homogeneous child node(best possible).
3. Extra Trees (Low Variance)
Extra Trees is like a Random Forest, in that it builds multiple trees and splits nodes using random subsets of features, but with two key differences: it does not bootstrap observations (meaning it samples without replacement), and nodes are split on random splits, not best splits. So in summary, ExtraTrees:
- builds multiple trees with bootstrap = False by default, which means it samples without replacement
- nodes are split based on random splits among a random subset of the features selected at every node
In Extra Trees, randomness doesn’t come from bootstrapping the data, but rather comes from the random splits of all observations. ExtraTrees is named for (Extremely Randomized Trees).