I have 100 classes in my input space. I plan to build 100 random forest classifiers that will determine if a given input matches a class (1) or not (0), such that a True/False model exists for each class. While optimizing these models, is it better statistical practice to optimize every model on the same (randomly generated) set of training and testing examples, or should I randomly select training and testing examples for each model separately? I want to avoid bias, so I am leaning towards random selection for each model separately, as I believe the result would be lower bias.

Edit: An advantage (in this scenario) of randomly selecting train/test sets for each model individually is that I can stratify my sampling for each separate model. This is useful, as each class only comprises a small percentage of the total training data. If I sample ahead of time, I can not do stratified selection for each class independently.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.