11
$\begingroup$

Is it fair to say Cross Validation (k-fold or otherwise) is unnecessary for Random Forest? I've read that is the case because we can look at out-of-bag performance metrics, and these are doing the same thing.

Please help me understand this in the context of RF for a classification problem. Thank you!

$\endgroup$
2
  • 5
    $\begingroup$ That's not true. Remember you fix certain parameters bef you grow a random forest (# trees and # of var to sample at each split). Cross validation helps you to pick these parameters. $\endgroup$
    – horaceT
    Jun 6, 2017 at 2:46
  • $\begingroup$ @horaceT: while you can auto-tune parameters using the oob estimate, you don't have to. And if you don't (and instead fix them yourself) then oob is as fair an estimate of generalization error as cross validation is. $\endgroup$ Jun 7, 2017 at 10:44

2 Answers 2

5
$\begingroup$

Yes, out-of-bag performance for a random forest is very similar to cross validation. Essentially what you get is leave-one-out with the surrogate random forests using fewer trees. So if done correctly, you get a slight pessimistic bias. The exact bias and variance properties will be somewhat different from externally cross validating your random forest.

Like for the cross validation, the crucial point for correctness (i.e. slight pessmistic bias, not large optistic bias) is the implicit assumption that each row of your data is an independent case. If this assumption is not met, the out-of-bag estimate will be overoptimistic (as would be a "plain" cross validation) - and in that situation it may be much easier to set up an outer cross validation that splits into independent groups than to make the random forest deal with such dependence structures.


Assuming you have this independence between rows, you can use the random forest's out-of-bag performance estimate just like the corresponding cross validation estimate: either as estimate of generalization error or for model tuning (the parameters mentioned by @horaceT or e.g. boosting). If you use it for model tuning, as always, you need another independent estimate of the final model's generalization error.

That being said, the no of trees and no of variables are reasonably easy to fix, so random forest is one of the models I consider with sample sizes that are too small for data-driven model tuning.

  • Prediction error will not increase with higher number of trees - it just won't decrease any further at some point. So you can just throw in a bit more computation time and be OK.

  • number of variates to consider in each tree will depend on your data, but IMHO isn't very critical, neither (in the sense that you can e.g. use experience on previous applications with similar data to fix it).

  • leaf size (for classification) is again typically left at 1 - this again doesn't cost generalization performance, but just computation time and memory.

$\endgroup$
2
  • $\begingroup$ I like to assure at least 5 leaf minimum per branch for classification, but you are dealing with small samples. When I am dealing with variable importance (Boruta) I like to fix at least 10 as minimum. I also like to, if I can, use binomial double trees (h2o) when I have that as an option. $\endgroup$ Jun 7, 2017 at 19:40
  • $\begingroup$ @EngrStudent: I sometimes also keep the leaf nodes somewhat bigger - I come across data with few independent cases but many repeated measurements. But then again, I fix it at larger node size - this is not something I'd subject to auto-tuning. $\endgroup$ Jun 8, 2017 at 17:22
5
$\begingroup$

Horace is right. When you think about OOB, you aren't exactly keeping it out of your model. Each tree may have different samples in-bag and out-of-bag but the forest uses them all. This allows information leakage.

The aggregation has its weaknesses. My understanding is that aggregation is accomplished as "weighted average" or "weighted vote". If there are two trees with equal "true" predictive ability over the population of true data, but one of them has higher effective accuracy over the training sample, then it will be given higher weight. The problem with that is, all else being equal, if it has higher accuracy on the training sample then it will have lower accuracy on values outside that sample, that is to say, in what is left of the general population. In this case OOB will have added error by how the bias impacts tree-weight in the aggregation.

For a single cross-validation then, imagine the same circumstance: two trees of equal "true" performance on the population, but one has better training results than the other. The holdback portion doesn't inform any of the trees, but it does allow unbiased estimation of performance. The testing error shows that the tree that had better "in-bag" performance will have equal to its counterpart on overall data, as long as the hold-out set is large enough. Classic OOB wouldn't show that.

References:

$\endgroup$
3
  • $\begingroup$ "My understanding is that aggregation is accomplished as "weighted average" or "weighted vote"." AFAIK plain "vanilla" random forests do just a plain average or majority vote (your choice). And the out-of-bag error is estimated on subsets which are in the end surrogate models like the CV surrogate models: out-of-bag error averages only prediction from trees that do not contain the case in question - and they are known to show a corresponding pessimistic bias (see your references). In-bag estimates are typically not even considered - trees are typically not pruned (=> 0% training error... $\endgroup$ Jun 7, 2017 at 10:48
  • $\begingroup$ ... for classification). You are right though that the kind of weighting you suggest would violate independence. $\endgroup$ Jun 7, 2017 at 10:52
  • 1
    $\begingroup$ @cbeleites - we both must look at source code. I should ask/answer a question on the topic of aggregation by package for R packages that do random-forest aggregation. :) Thank you. $\endgroup$ Jun 7, 2017 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.