Train-Test Splits in Random Forest approach with small sample sizes

Question

I am working with a small sample (n ~ 350) where I want to use a Random Forest approach to train a model to use on future new datapoints. I am using the randomForest package in R.

I understand the standard approach is to split the data into training and test data to validate the model. My concern lies with my small sample size. I also understand this concern could be assuaged using some type of k-fold validation.

However, why even perform the train-test split in the first place? Doesn't the Random Forest procedure bootstrap the data inherently? The randomForest package produces an MSE, which I understand is calculated using OOB predictions -- isn't this analogous to a train-test validation process? To me, it seems that this OOB MSE is analogous to a test MSE.

Sorry if I have a gross misunderstanding of the process here.

Answer 1

Yes, RFs' in-built OOB mse can be seen as an indicator for model performance. But you won't be able to compare its performance to different models (or different hyperparameters). Generally, you still want a "clean" hold-out set for validation.

Train-test splits are often quite inaccurate for small data sets. Consider the bootstrap or repeated k-fold CV, e.g. 100 times (stratified) 10-fold CV or similiar will probably be not too bad.