Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm trying to test random forest's ability to classify samples between 2 groups; There are 54 samples and varying numbers of variables used for classification.
I was wondering why the out-of-bag (OOB) estimates can vary as much as 5% from one another even when I'm using 50k trees? Is this something that bootstrapping could help with?
There are two sources of the OOB variance. One is the randomness of the procedure itself; this can be reduced by increasing the number of trees.
The other source of variance is the irreducible imperfection of having limited data and living in a complex world. Increasing the number of trees can't fix this.
Additionally, sometimes there just isn't enough data to solve the problem. For example, imagine two instances have the opposite labels but identical feature values. One of these samples will always be misclassified. (This is an extreme example, but illustrates how some problems are unfixable. We can relax it somewhat by considering a tiny perturbation to one vector; now it will usually be classified the same as its twin, but not always.) To solve this problem, you'd have to collect additional measurements to further distinguish the two points.
Increasing the number of trees can reduce the variance of the estimate of something like $p(y=1|x)$, though. Consider the results from the central limit theorem: increasing the sample size can reduce variance of a statistic like an average, but not eliminate it. The random forest predictions are an average $\bar{x}$ of all the trees' predictions, and these predictions are themselves random variables (because of the bootstrapping and random subsetting of features; both happen independently, so votes are also iid). The CLT provides that $\bar{x}$ approaches a normal distribution $\bar{x}\sim\mathcal{N}(\mu,\frac{\sigma^2}{n})$, where $\mu$ is the true mean prediction and $\sigma^2$ is the variance of the trees' votes. (Votes take values of either 0 or 1, so an average of the votes has finite variance.) The point is that doubling the number of trees will cut the variance of $\bar{x}$ in half, but won't drive it to zero. (Except when $\sigma^2=0$, but we know that is not the case here.)
Irreducible variance can't be fixed by bootstrapping. Moreover, random forests are already bootstrapped; it's part of the reason that it has "random" in its name. (The other reason being that a random subset of features are selected at each split.)
