After performing the following steps: 1) Split set into train and test 2) Do a grid search using k-fold cross validation on the training set to find the best hyper parameters

Can we use these hyper parameters and train now a model using both the training and the test set? Why? Is there any theoretical reference to look at?

Thank you in advance for your help!


The purpose of the test set is so you can verify, after cross validation to determine the best hyperparameters, that the model is probably going to generalize to new data. If seems to, then great.

If (1) you can expect the training data has the same statistical properties as the test set, and (2) so long as the model is somewhat stable (i.e. doesn't change its answers dramatically given different inputs), and (3) so long as the model parameters don't change dramatically with the addition of a few more examples (i.e. You didn't do any "early stopping".), then yes the hyperparameter settings you found with cross validation should also work well with a model trained on the whole set, and you might expect to even have a slightly better model by retraining on everything.

Randomly selecting testing points, assuming your data isn't a time-series and lends itself to this kind of separation, should usually satisfy point 1.

Point 2 can be satisfied by playing with the bias-variance tradeoff of your model.

For 3, ideally you have a large enough training set to begin with that the training process more or less converged, and training on the whole set does not improve the model very much more. If your training set was small enough that you really are concerned it did not converge, then parameter differences between a model trained on the smaller and larger sets could be substantial, which might render the hyperparameter choices found with cross-validation on the less-trained model poor choices for the more-trained model.

If any of these are violated, then retraining on the whole set before putting the thing out in to production is probably not a good idea, especially because you have no test-set sanity-check at the very end.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.