I want to choose a model class (e.g. logistic regression vs. random forests), but the validation set is used for selecting hyperparameters. Should I set aside a second validation set to select the model class?
My idea:
- Training set: choose parameters
- Validation set: choose hyperparameters
- Second validation set: choose model class (e.g. logistic regression vs. random forests)
- Test set: test model on unseen data
Or should I treat model class similarly as hyperparameters and select it based on the validation set performance?
Furthermore, we apply validation sets via cross-validation. Should I use a "nested" cross-validation to select the model class? A CV within a CV?