My post was inspired by this one (https://stackoverflow.com/questions/29290916/scikit-learn-random-forest-classifier-how-to-produce-a-plot-of-oob-error-agains)
Although random forest models do not overfit$^{1,2}$, it can be good to know how many trees are needed for a stable solution.
My question is about the best way to calculate OOB to know when additional trees no longer provide benefit (i.e. the OOB error is stable). As I understand it we have $B$ trees fit to $B$ many bagged datasets of size $N$, trying $\sqrt{p}$ predictors at each split. After fitting all of the trees we predict the class for every observation using only the trees for which it was withheld and then comparing it to the known class. This gives a misclassification rate for each tree and each observation (these become the predicted probabilities).
In R you can plot the sequential cumulative average of the OOB errors:
You can also do this in Python but requires a little more work since the trees are created sequentially$^{3,4}$.
It seems to me that when the number of trees in small (e.g. 1) there should be high variance in the OOB error. In other words if I fit 100 random forest models with 5 trees the average OOB error will have greater variance than if I fit 100 random forest models with 50 trees. Of course since we can fit the trees in parallel it is easy to grow many trees at once for a single model.
Does it make sense to calculate some sort of boosted OOB where the OOB score for $n$ trees is the average of $b$ samples of trees? For example take 10 samples of 5 trees out of the 500 trees and calculate the average OOB score from that. Then when the variance at number of trees $n$ is small stop growing trees (perhaps calculate something similar to the Gelman-Rubin statistic for MCMC convergence).
Since trees should be grown randomly rather than sequentially I'm not sure how much value there is to this, especially since I think the OOB error converges fairly quickly in most cases. However with larger datasets this may save some computational time when the number of trees needed is unknown.
Breiman states random forest models do not overfit: http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#remarks
but Mark R. Segal (April 14 2004. "Machine Learning Benchmarks and Random Forest Regression." Center for Bioinformatics & Molecular Biostatistics) says it can for noisy datasets (Does the optimal number of trees in a random forest depend on the number of predictors?)
