Random forest, cross validation or out-of-bag error?

Question

I am training a random forest on a text data set (that I represent with synthetic features) and I am willing to assess the quality of the features I am creating.

So far, I focused on the out-of-bag MSE. Alternatively, I could cross-validate the random forest. Though it seems more rigorous to use cross-validation, it will be (much) slower.

Is using the out-of-bag MSE a wrong approach for this task ? Are there any known biases ?

Answer 1

I don't know much of text analyses, so I cannot answer that part.

However, whatever CV metric favored in litterature should be compatible with OOB-CV also. Except that OOB-CV for time series permutes the series of events, I have not heard of any biases of using OOB-CV. As thumb rule, OOB-CV is similar to 5-fold CV.

Below I post a naive example for the R randomForest implementation of how to use other metrics.

library(randomForest)
set.seed(123)
obs=2000
X = matrix(rnorm(obs))
y = X+rnorm(obs)
plot(X,y)
rf = randomForest(X,y)
mse = function(pred,true) mean((pred-true)^2)
myCV_randomForest = function(rf,stat=mse,...) stat(predict(rf,...),rf$y)
myCV_randomForest(rf,stat=mse)
myCV_randomForest(rf,stat=function(x,y) sd(x-y)) #use some other metric

Answer 2

You have to be aware, that on average only exp(-1) * original_number_of_trees are used for the out-of-bag estimation of each observation, so you are evaluating a forest with less trees, which usually provide worse results than a forest with all trees (e.g. in cross_validation). You can avoid this by training a forest with more trees (e.g. multiplicate the number of wished trees by 1/exp(-1))

On the other hand it is more similar to leave-one-out than to e.g. 5-fold-CV, because on the OOB trees all observations except of the evaluated one are used.