I do certain classification task (machine learning, XGBoost), I do it according to kind of basic rules by splitting dataset into train/val/test sets (let's say 60%/20%/20%).

I choose the best model parameters depending how well it performs on val set.

Would it be beneficial to retrain said best model on train and val sets concatenated or even on all data train+val+test ? Any papers on that subject ? Any advantages/disadvantages ?


The short answer is yes.

Assuming there are no covariate shifts in your data (that is, samples in your dataset come from the same underlying distribution that does not change), having more data is always better. The reason we only use a fraction of the data for training and reserve the rest for validation/test sets is so that we can come up with the best procedure of fitting the model. Once we have obtained that procedure (including model choice, hyperparameter selection, ways to perform feature selection, outlier removal etc. - all of which we must do within the validation/cross-validation framework without cheating!), we should use that procedure to fit the model to all the data that we have, if we can (e.g. there are no computational constraints).

To identify how much you can expect to benefit from fitting your model to more data, you can plot a graph of your model performance vs the size of the training set used. It is possible that you will get very little from fitting the model to the whole dataset instead of only using 60% of it, but it is possible that it will improve the performance of the model significantly over your validation/test estimates.

If there are covariate shifts in your data, things become a bit more complicated. But generally, we would want to fit the model to the most recent data available in that case, paying less attention to older data which might have come from a very different distribution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.