How to handle hyperparameter tuning, cross-validation and the train/val/test split? I'm currently building a deep learning based model.
To tune my hyperparameters I'm using crossvalidation on the train/validation datasets. Thereby I should get a better estimate of how well my model can generalise.
Now, I'm left with the choice of how to train the final model that can be deployed. I thought of the following possibilities:


*

*Train on the full train/validation dataset and use the test set as "new" validation

*Train without test set validation – how can I know I'm not overfitting after n number of epochs?

*Use the best model from the initial (hyperparameter search) crossvalidation and train for longer

*Crossvalidation on the full train/validation/test dataset and select a model from there

*Other options?


In normal machine learning I would select option 2. – but in comparison to deep learning most models don't have the problem of overfitting as much.
I'm a bit confused on how to handle this problem and would love to get some of your input. Thanks in advance.
 A: Okay, cross-validation is a great starting point for hyperparameter tuning. There is no one right answer about this, but here are some general thoughts about your methods:
1. Train on the full train/validation dataset and use the test set as "new" validation.
I'm assuming this means that you train on the best hyperparameters, and test the resultant model on the test set. After that, no more hyperparameter changes. This is generally how final model scores are reported — or at least, how they should be.
2. Train without test set validation – how can I know I'm not overfitting after n number of epochs?
This is also a good sanity check that you should be doing in cross-validation anyway.
3. Use the best model from the initial (hyperparameter search) crossvalidation and train for longer.
This could potentially be a good step for your final model, provided that you have reason to believe that further training will improve bias and variance. You could simply do validation with a single hold-out dev set here as well and graph train vs. dev loss/accuracy across number of epochs.
4. Crossvalidation on the full train/validation/test dataset and select a model from there.
Ah, that's one of the cardinal sins of machine learning. DO NOT TOUCH YOUR TEST SET UNTIL YOU ARE 100% DONE WITH YOUR MODEL. DO NOT LOOK AT IT. DO NOT EVEN SNIFF IT. This is because any influence the test set has on the model defeats the purpose of generalization, because then you aren't really generalizing. Using the test set at any point in time in your development except when you are 100% done is...not good.
5. Other options?
Um, I would encourage you to do a Google search or search in some university course notes for this.
Also, NOT ALL HYPERPARAMETERS ARE EQUAL. Changing number of epochs vs. regularization parameter vs. learning rate may have drastically different effects on your generalization error, some of which you can apply general rules to, some of which you cannot. Just another thing to keep in mind :)
