2
$\begingroup$

Is it a viable solution to train on the whole dataset without splitting the data into 'train' and 'test' sets? In other words, is it okay to skip offline evaluation and only perform online evaluation (with the actual data that will occur in the future)? Wouldn't this provide my algorithm with more data, enabling it to derive conclusions better? After all, more data means better predictions, right?

I am using XGBoost with predefined hyperparameters, so there's no need for a 'validation' set.

I have been using an 80%:20% split, and my model never overfits. I don't see how offline evaluation provides me value.

EDIT: I am training on hundreds of millions of records on a lot of CPUs. I don't have the resources to retrain multiple times. So that is why my hyperparameters are predefined and that is why I skip cross-validation. Do you think that I can do something differently?

$\endgroup$
10
  • 3
    $\begingroup$ Welcome to Cross Validated! I have been using an 80%:20% split, and my model never overfits. How do you know you’ve not overfit? $\endgroup$
    – Dave
    Feb 27 at 12:43
  • 2
    $\begingroup$ It sounds as if you're saying you don't need to assess your model -- you are sure it is the best one for the situation. Then why ask the question? $\endgroup$
    – rolando2
    Feb 27 at 12:45
  • $\begingroup$ @Dave Multiple model iterations resulted in no significant change in performance, both offline and online. I hope I answered that right. I also edited my question and provided some more info. $\endgroup$
    – asparagus
    Feb 28 at 8:50
  • $\begingroup$ What do you mean by “multiple model iterations”? $\endgroup$
    – Dave
    Feb 28 at 11:22
  • $\begingroup$ I am training for 9 months. New data is coming in every day. Training every other day. There was no overfit ever because offline and online PR, RC, F1 and ROC_AUC are in the same ranges. $\endgroup$
    – asparagus
    Feb 28 at 12:24

2 Answers 2

1
$\begingroup$

I have been using an 80%:20% split, and my model never overfits.

You’ve already gotten value from doing the split. You used the split to show your model to be generalizable. If you’re using the predictions to make high-stakes decisions (e.g., betting your life’s savings), you can have confidence in those predictions. Without that confidence in the predictions, you almost might as well not have predictions.

An explicit out-of-sample set isn’t necessarily the way to go. There are arguments for multiple repeats of cross-validation or to bootstrap the model-building process, even to use a penalized measure of performance like adjusted $R^2$. The bootstrap procedure or a penalized measure would allow you to validate performance yet train on all data. Frank Harrell writes about this in his Regression Modeling Strategies and Biostatistics for Biomedical Research books.

However, something should penalize a predictive model for just connecting the dots and fitting to coincidences in the training data (noise) that will not be present in other data.

$\endgroup$
1
$\begingroup$

If you train on the entire dataset, you lose the ability to objectively assess your model's performance on new, unseen data. This could lead to a model that performs exceptionally well on your current data (since it has seen all of it) but fails to perform on future data because it has not been properly validated for generalization.

Even though you mentioned that your model has never overfitted with an $80\%:20\%$ split, the absence of a test set means there's no safeguard to check for overfitting in future changes to your model. Overfitting happens when the model learns the noise in the training data to the extent that it negatively impacts the performance on new data. Without a test set, you might not realize when your model starts to overfit.

Even if you're using predefined hyperparameters for XGBoost, there's always a possibility that these are not optimal for your specific dataset. Typically, hyperparameters are tuned on a validation set (or through cross-validation) to improve model performance. Without this step, you might be missing out on improved performance that could be achieved with better-tuned hyperparameters.

Solely relying on online evaluation can be risky, especially in a production environment. If the model performs poorly, it can lead to loss of revenue, customers, or credibility. Online evaluation should complement offline evaluation, not replace it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.