I perform multi-class classification using a Deep Neural Network. I split the data set into training, validation, and test set. To create a strong classificator, I train multiple models using various parameters (e.g. learning rate). Then I compare the results of each trained model on the test set, choosing the model that performed best.

Is this a valid approach? Or should choosing a model based on results on the test set be seen as an optimizing process, which might be biased towards the specific test set and thus not generalize well (overfitting)? Basically, am I violating the independence of the test set with this approach? If so, what would be the correct way to split the data with this approach?

  • $\begingroup$ You can use the test set to select your model, if you are indeed using the test set only to test the model. $\endgroup$ – user2974951 Sep 14 '18 at 12:47
  • $\begingroup$ If you use your validation set to select your model, you can keep your test set just for reporting on performance metrics $\endgroup$ – Dan Sep 14 '18 at 12:56

Using your test set for this purpose seems fundamentally wrong. Your algorithm choice is not conceptually different from tuning your hyper parameters. For example, a neural-net (NN) and a log-reg (LR) might be among your models to select; while a specific version of your neural-net (i.e. one neuron neural net with sigmoid activation func.) might correspond exactly to a log-reg in your hyper-parameter tuning process, where you advocate that there should be a validation set. Then, if you can use your test set for deciding between NN and LR; why can't you use them for hyper parameter tuning? This is why you should again have an independent test set.


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.