Is there a formula or rule for determining the correct sampSize for a randomForest?

Question

I'm playing with a randomForest and have found that generally increasing the sampSize leads to better performance. Is there a rule / formula / etc that suggests what the optimal sampSize should be or is it a trial and error thing? I guess another way of phrasing it; what are my risks of too small of a sampSize or too large (overfitting?)?

---

This question is referring to the R implementation of random forest in the randomForest package. The function randomForest has a parameter sampSize which is described in the documentation as

Size(s) of sample to draw. For classification, if sampsize is a vector of the length the number of strata, then sampling is stratified by strata, and the elements of sampsize indicate the numbers to be drawn from the strata.

Answer 1

In general, the sample size for a random forest acts as a control on the "degree of randomness" involved, and thus as a way of adjusting the bias-variance tradeoff. Increasing the sample size results in a "less random" forest, and so has a tendency to overfit. Decreasing the sample size increases the variation in the individual trees within the forest, preventing overfitting, but usually at the expense of model performance. A useful side-effect is that lower sample sizes reduce the time needed to train the model.

The usual rule of thumb for the best sample size is a "bootstrap sample", a sample equal in size to the original dataset, but selected with replacement, so some rows are not selected, and others are selected more than once. This typically provides near-optimal performance, and is the default in the standard R implementation. However, you may find in real-world applications that adjusting the sample size can lead to improved performance. When in doubt, select the appropriate sample size (and other model parameters) using cross-validation.

Answer 2

For random forests to work as well in new data as they do in training data, the required sample size is enormous, often being 200 times the number of candidate features. See here.

Answer 3

I ran 4500 random forests over night with some random parameter-settings:

Regression problem Ysignal = x1^2+sin(x2*pi) + x3 * x4 + x5 where any x are sampled independent from a normal distribution, sd=1, mean=1

Ytotal = Ysignal + Yerror

where Yerror = rnorm(n.observations,sd=sd(Ysignal))*noise.factor

theoretical.explainable.variance"TEV" = var(Ysignal= / var(Ytotal)

randomForest.performance = explained.variance(OOB cross-validation) / TEV

datasets were sampled from the regression problem and added noise n.obs was a random number between 1000 and 5000 n.extra.dummy.variables between 1 and 20

ntree always 1000

sample_replacement always true

mtry is 5 to 25, limited by n.obs noise.factor between 0 and 9

samplesize.ratio a random number between 10% and 100%, the ratio size of each bootstrap

all models were trained like rfo = randomForest(x=X, y=Ytotal, <more args>)

the randomForest.performance, its ability to explain the highest fraction of the TEV increases in general when samplesize lowers when the TEV is less than 50% and decrease when TEV is higher than 50%.

Thus, if your randomForest-modelfit reports e.g. 15% explained variance by OOB-CV, and this is an acceptable model-precision for you, then you can probably tweak the performance a little higher by lowering sampsize to a third of numbers of observations, given ntree > 1000.

Morale: For very noisy data it is better to de-correlate trees than to lower bias by growing maximal size trees.