10
$\begingroup$

I'm trying to test random forest's ability to classify samples between 2 groups; There are 54 samples and varying numbers of variables used for classification.

I was wondering why the out-of-bag (OOB) estimates can vary as much as 5% from one another even when I'm using 50k trees? Is this something that bootstrapping could help with?

$\endgroup$
  • 6
    $\begingroup$ You have to few samples. 50k trees doesn't make any sense with so few samples. The variation are most likely just one sample being incorrectly classified between runs. $\endgroup$ – ThiS Mar 22 '17 at 14:54
  • $\begingroup$ @ThiS I thought that increasing the number of trees would reduce the amount of variance I get. Is there a way to reduce it to effectively zero or know which one is the most accurate? $\endgroup$ – Sethzard Mar 22 '17 at 15:06
12
+50
$\begingroup$

There are two sources of the OOB variance. One is the randomness of the procedure itself; this can be reduced by increasing the number of trees.

The other source of variance is the irreducible imperfection of having limited data and living in a complex world. Increasing the number of trees can't fix this.

Additionally, sometimes there just isn't enough data to solve the problem. For example, imagine two instances have the opposite labels but identical feature values. One of these samples will always be misclassified. (This is an extreme example, but illustrates how some problems are unfixable. We can relax it somewhat by considering a tiny perturbation to one vector; now it will usually be classified the same as its twin, but not always.) To solve this problem, you'd have to collect additional measurements to further distinguish the two points.

Increasing the number of trees can reduce the variance of the estimate of something like $p(y=1|x)$, though. Consider the results from the central limit theorem: increasing the sample size can reduce variance of a statistic like an average, but not eliminate it. The random forest predictions are an average $\bar{x}$ of all the trees' predictions, and these predictions are themselves random variables (because of the bootstrapping and random subsetting of features; both happen independently, so votes are also iid). The CLT provides that $\bar{x}$ approaches a normal distribution $\bar{x}\sim\mathcal{N}(\mu,\frac{\sigma^2}{n})$, where $\mu$ is the true mean prediction and $\sigma^2$ is the variance of the trees' votes. (Votes take values of either 0 or 1, so an average of the votes has finite variance.) The point is that doubling the number of trees will cut the variance of $\bar{x}$ in half, but won't drive it to zero. (Except when $\sigma^2=0$, but we know that is not the case here.)

Irreducible variance can't be fixed by bootstrapping. Moreover, random forests are already bootstrapped; it's part of the reason that it has "random" in its name. (The other reason being that a random subset of features are selected at each split.)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.