8
$\begingroup$
  1. How many samples does each tree of a random forest use to train in sci-kit learn the implementation of Random Forest Regression?

  2. And, how does the number of samples change when the bootstrap option is on compared to when it’s off?

In “Random Forest” by Breiman, I believe he mentions that each tree is trained on 1/3 of the data. Is that the case in the implementation of scikit learn as well?

sklearn ref: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html

github source: https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/ensemble/forest.py#L1019

$\endgroup$
1
7
$\begingroup$

I am answering my question. I got a chance to talk to the people who implemented the random forest in sci-kit learn. Here is the explanation:

"If bootstrap=False, then each tree is built on all training samples.

If bootstrap=True, then for each tree, N samples are drawn randomly with replacement from the training set and the tree is built on this new version of the training data. This introduces randomness in the training procedure since trees will each be trained on slightly different training sets. In expectation, drawing N samples with replacement from a dataset of size N will select ~2/3 unique samples from the original set. "

From Scikit Learn v0.22, you can still use boostraping but limit the maximum number of samples each tree is trained on (max_samples of RandomForestRegressor class).

Excellent sources on this subject for more details:

$\endgroup$
3
  • $\begingroup$ But how do you set the sample size for each tree to be trained on? $\endgroup$ – verkter Sep 11 '19 at 22:32
  • $\begingroup$ Here's something weird: the default max_sample value is X.shape[0], i.e., the whole dataset. I'm contacting their forum to figure out why is being used as default the whole dataset. $\endgroup$ – Fernando Wittmann May 8 '20 at 21:20
  • $\begingroup$ sorry, I'm not clear on your question. Usually, X.shape[0] should be your total number of samples. Bootstrapping is usually sampling with replacement where the unique number of samples can be estimated as explained above. So by default: the bag size of your sampling with replacement = X.shape[0] = total number of samples $\endgroup$ – sinapan May 8 '20 at 23:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.