When to *not* split up your data into training and testing

Question

So, I was thinking of a situation of when to not split up your data into training and testing and to just train on the entire dataset, at the risk of "overfitting". If my dataset has let's say 10 columns of which 9 are categorical (which have no particular order, and one is numeric). It could happen that your training set split may not capture observations that include some of the original categorical observations. When you then go and try to evaluate your model on the test set, which will have these new categorical variables that were not trained on, your model would potentially not know what to do, and in R, or wherever your model is implemented, you would get an error.

Another situation could be where you are not interested in predicting "new data", rather, just getting a feel for the relationships/patterns between the predictors and the response variable.

Would these not be situations that even though your model will potentially be overfitted, that you would want to train on the entire dataset?

Thanks!

Answer 1

I agree with your first point: if the amount of data is limited, then we can use the entire data for model building, instead of split the data into two sets.

However, I do not completely agree with your second point: even there are no new data to predict, and all we want is trying to discover patterns. Sometimes testing data is still needed.

For example, we want to use K-means to "discover patterns" in data. How many clusters to choose can be derived using a testing data set.

Always keep in mind, data can contain noise, and if we just do whatever we can to overfit one data set. The discovery can be false and not generalized well.

Answer 2

You don't need to do a train/validation (or test) split, if what you do cannot be evaluated on a bit of data you did not use. Clustering could be an example, the assessment of clustering is often just by human gut feeling (e.g. humans might hope that similar countries/products/whatever get clustered together). However, if the next step is to use the clusters as part of some predictive algorithm, this may again need splitting.

Regarding the example with categories that may not be present in the training part of a split, I think the comment by @Glen_b is one good perspective on this (i.e. split so that's not an issue). Additionally, you could also make sure your approach can handle this type of data situation (for which there are many approaches depending on the exact setting), which is particularly valuable, if even more categories (for which you have no data, so far) may "turn up" in the future. In that case, splitting gives you an evaluation of how well what you are doing performs when faced with a new previously unseen category.

If anything, a small data setting might be the setting, where you really need a robust evaluation approach, because it's so incredibly easy to overfit a small dataset. A single train-test split is probably not it though, cross-validation or perhaps overfitting-corrected bootstrapping might be better. E.g. Harrell mentions in Section 5 of his Regression Modelling Stratgies book as a good internal (=when we cannot test on large new external data, or will only do so later) validation strategy:

Strong internal validation using 100 repeats of 10-fold cross-validation or several hundred bootstrap resamples, repeating all analysis steps involving Y afresh at each re-sample and the arbitrariness of selected 'important variables' is reported (if variable selection is used)

Answer 3

Would like to add my comments to the second point.

In the situation that we are using a regression model to discover pattern in the data, I would think that we do not need a training, testing and validation data set to select the model. There are other methods available to assist us in eliminating "noise" to prevent over-fitting. For instance, we can use metrics such as AIC, leave one out cross-validation (jackknife) or adjusted R-square. Using these metrics above, it will help us to eliminate the possibility of overfitting and the aim of interpolation as oppose to using a testing and validation set which is more related to extrapolation.

Feel feel to add in your views on this.

Answer 4

Regarding your second point, if you are referring to clustering algorithms, then you do not split the data into train and test. That is because we are not predicting or classifying anything and so we do not need the test or validation set. We train the clustering algorithm on the full dataset.