Is it necessary to do $k$-fold cross validation for decision trees in random forests?

Question

Consider the following data set train:

    z   a   b   c
    0   1   40  185
    0   1   25  128
    0   0   32  100
    0   0   29  100
    1   1   30  107
    0   0   30  133
    1   1   38  132
    1   1   37  127
    1   0   30  184
    1   0   40  199
    1   1   26  185
    0   1   21  185
    0   0   21  134
    0   0   20  137
    1   1   22  135
    0   0   23  189
    1   0   32  109
    1   0   31  152
    1   0   38  130
    1   1   37  191
    0   1   39  168
    1   0   28  183
    0   1   26  171
    1   1   23  164
    0   1   32  111
    0   0   34  131
    1   0   30  121
    1   0   27  195
    1   1   29  117
    1   0   26  187
    1   0   34  183
    0   0   28  189
    0   1   34  150
    0   1   34  176
    0   1   24  140
    1   0   37  181
    0   1   36  109
    1   0   39  198
    0   0   32  164

where z is a binary variable with predictors a,b,c. Suppose that there is some other test set with the same variables as the train data set and we want to predict z. For decision trees, is it better to use the full train data set to construct the tree? What would the purpose of $4$-fold cross validation be for example?

In a random forest, is $k$-fold cross-validation necessary? I thought you could use OOB error?

Answer 1

I can recommend this article dicussing good CV practice.

• (A) When simply running one RF model: Yes OOB-CV is a fine estimate of your future prediction performance, given i.i.d. sampling. For many practical instances you don't have time nor need for anything more. A default RF model is simply good enough, you will first start fiddling with the hyper parameters later, if ever. I would spent more time wondering which prediction performance metric(AUC, accuracy, recall etc) best answered my questions. A little fiddling with hyperparameters (mtry,samplesize), does not make your OOB-CV vastly over-optimistic.

• (B1) When comparing across classifiers (SVM, logistic, RF, etc.): you need to use the same CV regime, thus you cannot use OOB-CV only available for RF. Use e.g. 20-repeated, 10-fold CV, where all models are tested in the same folds(/by same partitions).

• (B2) When performing gridSearch and variable selection To evaluate the predictive performance of each variant of your model you would probably use OOB-CV or some other CV. To unbiased estimate the overall performance, you need to wrap your model-selection process in a outer cross-validation, called nested-CV.

Answer 2

For decision trees, is it better to use the full train data set to construct the tree?

It is always better to have more data to train your model. But if you use all data that you have in hand, then you have no idea about your test error (of course you can indirectly estimate it but estimations remain estimations), and furthermore, it is hard to know if you overfit your data or not. So in my experience, it is not recommended to use all data to fit your model.

In a random forest, is k-fold cross-validation necessary? I thought you could use OOB error?

OOB error can be used to tune your parameters, once it's done, OOB is no longer a valid test set for evaluating your model.

Answer 3

The answer is No. Random forests don't need k-fold CV. As an example, when I compare RF classification results with other classifiers for which k-fold CV was used, I only perform a single run with the entire input dataset for the RF model. RF will take care of the training and testing data by itself -- so it is unlike many other classifiers, and this is by design (by Leo Breiman).