4
$\begingroup$

I am training a random forest on a text data set (that I represent with synthetic features) and I am willing to assess the quality of the features I am creating.

So far, I focused on the out-of-bag MSE. Alternatively, I could cross-validate the random forest. Though it seems more rigorous to use cross-validation, it will be (much) slower.

Is using the out-of-bag MSE a wrong approach for this task ? Are there any known biases ?

$\endgroup$
3
  • 2
    $\begingroup$ And don't forget the Efron-Gong optimism bootstrap- a bias estimator for amount of overfitting that you subtract from the apparent accuracy. $\endgroup$ Feb 3, 2016 at 12:25
  • $\begingroup$ @FrankHarrell thanks ! could you please elaborate for those (like me) who are not familiar with the Efron-Gong optimism bootsrap ? $\endgroup$
    – RUser4512
    Feb 3, 2016 at 13:43
  • 3
    $\begingroup$ This is the original bootstrapping approach to estimating likely forecast accuracy in new samples without waiting for new samples. It is what is implemented in the R rms package validate and calibrate functions. $\endgroup$ Feb 3, 2016 at 17:11

2 Answers 2

1
$\begingroup$

I don't know much of text analyses, so I cannot answer that part.

However, whatever CV metric favored in litterature should be compatible with OOB-CV also. Except that OOB-CV for time series permutes the series of events, I have not heard of any biases of using OOB-CV. As thumb rule, OOB-CV is similar to 5-fold CV.

Below I post a naive example for the R randomForest implementation of how to use other metrics.

library(randomForest)
set.seed(123)
obs=2000
X = matrix(rnorm(obs))
y = X+rnorm(obs)
plot(X,y)
rf = randomForest(X,y)
mse = function(pred,true) mean((pred-true)^2)
myCV_randomForest = function(rf,stat=mse,...) stat(predict(rf,...),rf$y)
myCV_randomForest(rf,stat=mse)
myCV_randomForest(rf,stat=function(x,y) sd(x-y)) #use some other metric
$\endgroup$
1
$\begingroup$

You have to be aware, that on average only exp(-1) * original_number_of_trees are used for the out-of-bag estimation of each observation, so you are evaluating a forest with less trees, which usually provide worse results than a forest with all trees (e.g. in cross_validation). You can avoid this by training a forest with more trees (e.g. multiplicate the number of wished trees by 1/exp(-1))

On the other hand it is more similar to leave-one-out than to e.g. 5-fold-CV, because on the OOB trees all observations except of the evaluated one are used.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.