Do I need a second validation set to select model class?

Question

I want to choose a model class (e.g. logistic regression vs. random forests), but the validation set is used for selecting hyperparameters. Should I set aside a second validation set to select the model class?

My idea:

• Training set: choose parameters
• Validation set: choose hyperparameters
• Second validation set: choose model class (e.g. logistic regression vs. random forests)
• Test set: test model on unseen data

Or should I treat model class similarly as hyperparameters and select it based on the validation set performance?

Furthermore, we apply validation sets via cross-validation. Should I use a "nested" cross-validation to select the model class? A CV within a CV?

Answer 1

Prior to the revival of deep learning in the last few years, hyperparameter tuning used to be called model selection. The purpose of the validation set is to choose among several model candidates. It shouldn't make a difference whether these models have the same architecture with different hyperparameters or are completely different architectures.

So no, you shouldn't need a second validation set.

Answer 2

I am actually doing this right now too! :) I have 3 model classes, logistic, random forest and GP.

My design is this (with 5-fold crossvalidation):

• training data set - optimize parameters and hyperparameters (not sure if we have the same definition of hyperparameters; in my case these are the length-scales for GP covariance matrix).

• validation data set - cross-validate models and compare them within & between classes using common test statistics

I suppose this should be perfectly OK, if you have any ideas why this could be a problem let us discuss it.