10
$\begingroup$

As far as I've seen, opinions tend to differ about this. Best practice would certainly dictate using cross-validation (especially if comparing RFs with other algorithms on the same dataset). On the other hand, the original source states that the fact OOB error is calculated during model training is enough of an indicator of test set performance. Even Trevor Hastie, in a relatively recent talks says that "Random Forests provide free cross-validation". Intuitively, this makes sense to me, if training and trying to improve a RF-based model on one dataset.

What's your opinion on this?

$\endgroup$
  • 3
    $\begingroup$ this is not addresssing the main point of the question - but you still would probably want to cross-validate secondary parameters (such as depth of trees, etc.) $\endgroup$ – Wouter Jul 20 '15 at 21:34
  • $\begingroup$ You can use RF or compare it to other approaches in terms of performance on the training set, or use independent/subset of data to test the performance. It is a question of your hypothesis: are you trying to generalize the results to a larger population or just to classify the data at hand, rather than a property of RF. $\endgroup$ – katya Jul 20 '15 at 22:29
3
$\begingroup$

The OOB error is calculated by for each observation using only the trees that did not have this particular observation in their bootstrap sample; see this related question. This is very roughly equivalent to two-fold cross validation as the probability of a particular observation being in a particular bootstrap sample is $1-(1-\frac{1}{N})^N \approx 1-e^{-1} \approx 0.6$.

As @Wouter points out, you will probably want to do cross validation for parameter tuning, but as an estimate of test set error the OOB error should be fine.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.