2
$\begingroup$

This question already has an answer here:

I have heard that the main reason to use out-of-bag sample(OBB) over CV is that OOB is faster to implement and usually gets the same results(for tuning the hyper-parameters) as implementing CV when the Random forest model is used.

If this isn't true, in which cases using CV gives better results?

$\endgroup$

marked as duplicate by cbeleites, mdewey, Peter Flom Jul 20 '17 at 13:02

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

2
$\begingroup$

Bootstrapping and CV have tradeoffs:

  • Bootstrapping can be used to generate an enormous number of "new" datasets which are "noisy" versions of the original dataset (in the sense that the samples are weighted differently than in the original dataset). However, each bootstrap sample sees approximately 2/3 of the samples, irrespective of the dataset size.

  • Cross validation allows controlling more precisely the number of samples used. In 10-fold cross validation, for example, each iteration will train on (ignoring roundoff errors) approximately 90% of the data.

Bootstrapping, therefore, is useful for things like training many high-variance low-bias predictors (like the trees in a random forest), or estimating confidence intervals of a statistic using many iterations. Conversely, it can underestimate the actual performance of the predictor.

As you pointed out in your excellent comment below, Breiman indeed states that the OOB error rate is a good estimator for OOS error rate. With all enormous respect due to Breiman, FWIW, I've seen different results, especially for datasets with very low SNR. Some of the answers to this question state this too. I assume that, asymptotically, for a given SNR, if the datasets grow large enough, Breimann is correct; in practice, you might want to be cautious about it.

$\endgroup$
  • $\begingroup$ Regarding the speed issue...Bootstrapping is already needed to fit each individual tree. So, using the out-of-bag samples to estimate performance can be done 'for free', rather than re-fitting the forest k times for k-fold cross validation. $\endgroup$ – user20160 Jul 19 '17 at 6:58
  • $\begingroup$ @user20160 I don't think so. Bootstrapping is used to fit the tree using the OOB samples, and so cannot be used to estimate the performance. That would be an overestimation of the performance of the tree. Conversely, using CV for the forest, and within each fold using boosting, is OK. $\endgroup$ – Ami Tavory Jul 19 '17 at 7:05
  • 2
    $\begingroup$ Perhaps we're talking about different things? Individual trees are trained on in-bag points, whereas out-of-bag points can be used to estimate error. That's what Breiman himself recommends here. Also mentioned in the chapter on RFs in Hastie&Tibshirani Elements of Statistical Learning. Of course, if using OOB points for hyperparameter tuning, an outer CV loop would be needed to estimate performance. $\endgroup$ – user20160 Jul 19 '17 at 9:03
  • $\begingroup$ @user20160 That is an excellent point (I assumed you meant that OOB is used to fit hyperparameters). Have updated the answer to address it. $\endgroup$ – Ami Tavory Jul 19 '17 at 9:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.