What are best practices around combining train/val/test splits when training a production model?

Question

Say I'm training a deep neural network and I have split my data into train, val, and test splits. I have trained many models on the train data and then using the val data during the training loop for things like ReduceLRonPlateau and early stopping. Then I test all my models on the test set and determine that one of my models performs the best. Then, I would like to use the hyperparameters I used in my best model to train a production model and I would like to use as much of the data as I can. I would like to understand the best practices around doing this.

One option would be to combine all the data into one set and train it using the learning rate schedule that my previous ReduceLRonPlateau produced and stopping it at the same number of epochs. But since I have more data this time, I would think a few extra epochs might help. Is this a bad idea?

Another option would be to combine the train and test sets and then use the val set again for ReduceLRonPlateau and early stopping. This way the training process is still based on my dataset. Is this a good idea? Is there a risk that my result would be worse than my model that was only based on the train split? Are there better options out there?

Answer 1

Then I test all my models on the test set and

Whoa, let me stop you right there. The test set is used once and only once to get a final estimate of the out of sample performance. If you use the test set for model selection, it is no longer a test set -- it is another validation set. In general, any data you use to change your mind about a model is a part of the validation set. I would recommend you select models based on validation loss and leave the test set to be used only as a final estimate of out of sample performance.

One option would be to combine all the data into one set and train it using the learning rate schedule that my previous ReduceLRonPlateau produced and stopping it at the same number of epochs. But since I have more data this time, I would think a few extra epochs might help. Is this a bad idea?

Not bad per se, but now you've constructed a model which is unvalidated. In the training tep, you've validated the model using the validation set, and get a final estimate of performance from the test set. That final estimate is applicable to the model trained on the training set (hence the name, training set). To add additional data or to train for longer (more epochs) is a different model (although would probably perform the same, perhaps a bit worse perhaps a bit better. Who is to say? You don't have a validation set so you can't say for sure).

Now, this sort of approach where model selection is done and then a new model is trained on the entire data with the optimal parameters is done a lot. It is usually harmless, but a principled approach would see the validated model productionalized, not a model trained on more data with the optimal parameters determined from a subset of the entire data.

Answer 2

There are a number of issues that need to be addressed before considering best practices (and the issues below could be, in part, components of best practice).

1. Are you ensuring that each object in the entire data set gets a chance to be used in testing (while left out of training) at least once, i.e., via CV?
2. Since ANNs can also learn based on the order of training objects presented, are you permuting all objects (using random shuffling) a second, third, ..., tenth time before assigning all of them to the training, validation, and test folds, and running CV through all the test folds again?
3. ANNs will also learn the correlation between features, which wastes time and is inefficient. How much correlation is there between features used for training(testing)?

Predictive error can be tracked for each object as it is tested if needed, otherwise the confusion matrix can be incremented over the entire run, and accuracy(error) determined thereafter.