8
$\begingroup$

I want to choose a model class (e.g. logistic regression vs. random forests), but the validation set is used for selecting hyperparameters. Should I set aside a second validation set to select the model class?

My idea:

  • Training set: choose parameters
  • Validation set: choose hyperparameters
  • Second validation set: choose model class (e.g. logistic regression vs. random forests)
  • Test set: test model on unseen data

Or should I treat model class similarly as hyperparameters and select it based on the validation set performance?

Furthermore, we apply validation sets via cross-validation. Should I use a "nested" cross-validation to select the model class? A CV within a CV?

$\endgroup$

2 Answers 2

8
$\begingroup$

Prior to the revival of deep learning in the last few years, hyperparameter tuning used to be called model selection. The purpose of the validation set is to choose among several model candidates. It shouldn't make a difference whether these models have the same architecture with different hyperparameters or are completely different architectures.

So no, you shouldn't need a second validation set.

$\endgroup$
1
  • $\begingroup$ +1 exactly what I thought, I would just use training data set for hyperparameters... maybe we are just using the term "hyperparameters" for different thing :-) See my answer. $\endgroup$
    – Tomas
    Commented Nov 8, 2019 at 19:54
0
$\begingroup$

I am actually doing this right now too! :) I have 3 model classes, logistic, random forest and GP.

My design is this (with 5-fold crossvalidation):

  • training data set - optimize parameters and hyperparameters (not sure if we have the same definition of hyperparameters; in my case these are the length-scales for GP covariance matrix).

  • validation data set - cross-validate models and compare them within & between classes using common test statistics

I suppose this should be perfectly OK, if you have any ideas why this could be a problem let us discuss it.

$\endgroup$
8
  • $\begingroup$ I guess one clarifying point is when you said "choose hyperparameters" in the training set. To be clear, within a training set you should set your hyperparameters and choose parameters via MSE or whatever. Do this over all model-class/hyperparameter combinations. Then, for each combo, measure out-of-sample performance on the validation set. Finally, my idea for comparing model classes is to select the model within each class with the best out-of-sample performance (i.e. hyperparams gave best validation performance). Then compare these class-best models between classes. What do you think? $\endgroup$ Commented Nov 8, 2019 at 20:05
  • $\begingroup$ " To be clear, within a training set you should set your hyperparameters and choose parameters" - I am still not understanding what you mean by this. What is the difference between set and choose? What I am doing actually is to optimize these (hyper)parameters with L-BFGS optim(), which I guess is perfectly OK with the training set. $\endgroup$
    – Tomas
    Commented Nov 8, 2019 at 20:09
  • $\begingroup$ ... i.e., it's not clear to me why for an optimization of hyperparameters with L-BFGS would I use the validation dataset. $\endgroup$
    – Tomas
    Commented Nov 8, 2019 at 20:11
  • $\begingroup$ Sorry, I was unclear. You optimize parameters on a training set, and optimize hyperparameters using some validation set. You cannot (should not?) optimize both on one set. If you're using CV within the "training" set to optimize hyperparameters, then you're good. $\endgroup$ Commented Nov 8, 2019 at 20:13
  • $\begingroup$ @RobertHatem you are mixing things up totally now :-) 1) "You optimize parameters on a training set, and optimize hyperparameters using some validation set." no! I told you above, I am using training set for optimizing both! :-) 2) "You cannot (should not?) optimize both on one set." why? 3) " If you're using CV within the "training" set to optimize hyperparameters" no. I am using the CV with the validation set for validating (second point in my answer), please check my answer. $\endgroup$
    – Tomas
    Commented Nov 8, 2019 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.