I am working with a small sample (n ~ 350) where I want to use a Random Forest approach to train a model to use on future new datapoints. I am using the randomForest package in R.

I understand the standard approach is to split the data into training and test data to validate the model. My concern lies with my small sample size. I also understand this concern could be assuaged using some type of k-fold validation.

However, why even perform the train-test split in the first place? Doesn't the Random Forest procedure bootstrap the data inherently? The randomForest package produces an MSE, which I understand is calculated using OOB predictions -- isn't this analogous to a train-test validation process? To me, it seems that this OOB MSE is analogous to a test MSE.

Sorry if I have a gross misunderstanding of the process here.


1 Answer 1


Yes, RFs' in-built OOB mse can be seen as an indicator for model performance. But you won't be able to compare its performance to different models (or different hyperparameters). Generally, you still want a "clean" hold-out set for validation.

Train-test splits are often quite inaccurate for small data sets. Consider the bootstrap or repeated k-fold CV, e.g. 100 times (stratified) 10-fold CV or similiar will probably be not too bad.

  • $\begingroup$ Thanks for your answer -- that's what I suspected but I wanted to be sure. It's fine if I can't compare to different models. I have a specific use for the Random Forest (I also want to make use of the Proximity Matrix). $\endgroup$
    – smsarkar
    May 4, 2021 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.