10
$\begingroup$

As far as I've seen, opinions tend to differ about this. Best practice would certainly dictate using cross-validation (especially if comparing RFs with other algorithms on the same dataset). On the other hand, the original source states that the fact OOB error is calculated during model training is enough of an indicator of test set performance. Even Trevor Hastie, in a relatively recent talks says that "Random Forests provide free cross-validation". Intuitively, this makes sense to me, if training and trying to improve a RF-based model on one dataset.

What's your opinion on this?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ this is not addresssing the main point of the question - but you still would probably want to cross-validate secondary parameters (such as depth of trees, etc.) $\endgroup$
    – Wouter
    Commented Jul 20, 2015 at 21:34
  • $\begingroup$ You can use RF or compare it to other approaches in terms of performance on the training set, or use independent/subset of data to test the performance. It is a question of your hypothesis: are you trying to generalize the results to a larger population or just to classify the data at hand, rather than a property of RF. $\endgroup$
    – katya
    Commented Jul 20, 2015 at 22:29

1 Answer 1

Reset to default
4
$\begingroup$

The OOB error is calculated by for each observation using only the trees that did not have this particular observation in their bootstrap sample; see this related question. This is very roughly equivalent to two-fold cross validation as the probability of a particular observation being in a particular bootstrap sample is $1-(1-\frac{1}{N})^N \approx 1-e^{-1} \approx 0.6$.

As @Wouter points out, you will probably want to do cross validation for parameter tuning, but as an estimate of test set error the OOB error should be fine.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.