Why do I keep reading about specifying training and test sets with random forest regression? As probably obvious from this question I am new to this method, but what I thought one of the cool things about it was that this is not needed because the out of bag sample functions as the test set, i.e. the method provides its own test set. Is this just needed under specific circumstances, e.g. for more methodological studies or am I missing some very fundamental aspect?? Thank you for any clarification.

  • $\begingroup$ Dividing the data into two sets is quite wasteful of sample size (as opposed to using rigorous bootstrap validation) and requires an initial sample size of at least n=20,000 to be stable/reliable. $\endgroup$ – Frank Harrell May 10 '16 at 15:24
  • $\begingroup$ Thank you both. I know it was a horribly vague question, but these answers are all I needed. I don't have a specific paper to talk about, I just saw the topic of test/training samples come up everywhere, including in the options of R random forest packages and was afraid I was missing something. Frank, do you maybe have a reference for that sample size statement? $\endgroup$ – Johannes May 12 '16 at 1:17
  • $\begingroup$ I don't have a reference but base this on two things: (1) A dataset with n=17,000 and proportion $Y=1$ of 0.3 resulted in unstable $c$-index (ROC area) when using split-sample validation twice. (2) Single 10-fold cross-validation has insufficient precision; you need to repeat 50-100 times for stability. And a single 10-fold cross-validation is much better than 2-fold cv. $\endgroup$ – Frank Harrell May 12 '16 at 11:42

You still need a separate test set (or outer round of cross-validation) if you're using those out-of-bag estimates to tune some aspect of the random forest, such as the number of features to consider for each split. If not, out-of-bag estimates can indeed substitute for a separate test set.

I may be able to give you a more specific answer if you provide the particular paper or papers you're talking about.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.