Can I skip test set and train on 100% of data?

Question

Is it a viable solution to train on the whole dataset without splitting the data into 'train' and 'test' sets? In other words, is it okay to skip offline evaluation and only perform online evaluation (with the actual data that will occur in the future)? Wouldn't this provide my algorithm with more data, enabling it to derive conclusions better? After all, more data means better predictions, right?

I am using XGBoost with predefined hyperparameters, so there's no need for a 'validation' set.

I have been using an 80%:20% split, and my model never overfits. I don't see how offline evaluation provides me value.

EDIT: I am training on hundreds of millions of records on a lot of CPUs. I don't have the resources to retrain multiple times. So that is why my hyperparameters are predefined and that is why I skip cross-validation. Do you think that I can do something differently?

Answer 1

I have been using an 80%:20% split, and my model never overfits.

You’ve already gotten value from doing the split. You used the split to show your model to be generalizable. If you’re using the predictions to make high-stakes decisions (e.g., betting your life’s savings), you can have confidence in those predictions. Without that confidence in the predictions, you almost might as well not have predictions.

An explicit out-of-sample set isn’t necessarily the way to go. There are arguments for multiple repeats of cross-validation or to bootstrap the model-building process, even to use a penalized measure of performance like adjusted $R^2$. The bootstrap procedure or a penalized measure would allow you to validate performance yet train on all data. Frank Harrell writes about this in his Regression Modeling Strategies and Biostatistics for Biomedical Research books.

However, something should penalize a predictive model for just connecting the dots and fitting to coincidences in the training data (noise) that will not be present in other data.

Answer 2

If you train on the entire dataset, you lose the ability to objectively assess your model's performance on new, unseen data. This could lead to a model that performs exceptionally well on your current data (since it has seen all of it) but fails to perform on future data because it has not been properly validated for generalization.

Even though you mentioned that your model has never overfitted with an $80\%:20\%$ split, the absence of a test set means there's no safeguard to check for overfitting in future changes to your model. Overfitting happens when the model learns the noise in the training data to the extent that it negatively impacts the performance on new data. Without a test set, you might not realize when your model starts to overfit.

Even if you're using predefined hyperparameters for XGBoost, there's always a possibility that these are not optimal for your specific dataset. Typically, hyperparameters are tuned on a validation set (or through cross-validation) to improve model performance. Without this step, you might be missing out on improved performance that could be achieved with better-tuned hyperparameters.

Solely relying on online evaluation can be risky, especially in a production environment. If the model performs poorly, it can lead to loss of revenue, customers, or credibility. Online evaluation should complement offline evaluation, not replace it.