Can you include all your variables in a Random Forest at once?

Question

As I understand, there are two sources of variations when growing the decisions trees of a random forest: one from the bootstrapping and another from the random selection of the variables available in your dataset that will be used to construct each tree (is this called feature section, by the way?)

Let's say I have four "independent" variables in my dataset, the Random Forest algorithm should not (or cannot?) use all of them to grow all the decision trees, but rather a random selection of, for example, two of those variables for each tree.

Is that right? Can Random Forests ever include all available variables to grow each tree or does this random variable selection always need to be present?

Answer 1

Random forest does the random sampling from among features at each split within a decision tree. At each split, random forest randomly selects a small number of features from among all features that you provide. From this subset, the feature that provides the best improvement to the model is used at that split.

Whether or not all features are used depends on the data and the configuration settings for the random forest. There's two reasons why a feature might not be used:

• The vagaries of random sampling mean that the feature was never selected for a split. This is unlikely when the number of total splits in the ensemble is large relative to the number of features.
• A feature has no information about the target variable. An example of this is a feature that takes a constant value.

If you don't want random forest to use a feature to make a split, then don't provide it to the model.

Detailed explanations of the random forest procedure and its statistical properties can be found in Leo Breiman, "Random Forests," Machine Learning volume 45 issue 1 (2001) as well as the relevant chapter of Hastie et al., Elements of Statistical Learning.