Random forest, cross validation or out-of-bag error?

I am training a random forest on a text data set (that I represent with synthetic features) and I am willing to assess the quality of the features I am creating.

So far, I focused on the out-of-bag MSE. Alternatively, I could cross-validate the random forest. Though it seems more rigorous to use cross-validation, it will be (much) slower.

Is using the out-of-bag MSE a wrong approach for this task ? Are there any known biases ?

• And don't forget the Efron-Gong optimism bootstrap- a bias estimator for amount of overfitting that you subtract from the apparent accuracy. – Frank Harrell Feb 3 '16 at 12:25
• @FrankHarrell thanks ! could you please elaborate for those (like me) who are not familiar with the Efron-Gong optimism bootsrap ? – RUser4512 Feb 3 '16 at 13:43
• This is the original bootstrapping approach to estimating likely forecast accuracy in new samples without waiting for new samples. It is what is implemented in the R rms package validate and calibrate functions. – Frank Harrell Feb 3 '16 at 17:11

I don't know much of text analyses, so I cannot answer that part.

However, whatever CV metric favored in litterature should be compatible with OOB-CV also. Except that OOB-CV for time series permutes the series of events, I have not heard of any biases of using OOB-CV. As thumb rule, OOB-CV is similar to 5-fold CV.

Below I post a naive example for the R randomForest implementation of how to use other metrics.

library(randomForest)
set.seed(123)
obs=2000
X = matrix(rnorm(obs))
y = X+rnorm(obs)
plot(X,y)
rf = randomForest(X,y)
mse = function(pred,true) mean((pred-true)^2)
myCV_randomForest = function(rf,stat=mse,...) stat(predict(rf,...),rf\$y)
myCV_randomForest(rf,stat=mse)
myCV_randomForest(rf,stat=function(x,y) sd(x-y)) #use some other metric

You have to be aware, that on average only exp(-1) * original_number_of_trees are used for the out-of-bag estimation of each observation, so you are evaluating a forest with less trees, which usually provide worse results than a forest with all trees (e.g. in cross_validation). You can avoid this by training a forest with more trees (e.g. multiplicate the number of wished trees by 1/exp(-1))

On the other hand it is more similar to leave-one-out than to e.g. 5-fold-CV, because on the OOB trees all observations except of the evaluated one are used.