Perform cross-validation on train set or entire data set

Question

I am a bit confused concerning how I should perform cross-validation to evaluate a statistical learning model.

If I have a data set of 500 observations, should i divide it in a train and test set with for example 375 (75%) train observations and 125 (25%) test observations, and perform cross-validation on the train set? Or should I perform the cross-validation on the entire data set?

So long as the aim of performing cross-validation is to acquire a more robust estimate of the test MSE, and not to optimize some tuning parameter, my understanding is that you should use the entire data set. The reason for this is that you would not acquire a model that you can use to predict on unseen test observations, just a measure of MSE for the train set where cross-validation is performed.

If I am mistaken, how could I use the cross-validation result to predict out of sample observations?

Could someone be kind and clarify this for me?

If relevant, the problem I am solving is performing cross-validation to assess the model performance of a random forest model in R. Thanks in advance!

Answer 1

should i divide it in a train and test set with for example 375 (75%) train observations and 125 (25%) test observations, and perform cross-validation on the train set?

Yes

Or should I perform the cross-validation on the entire data set?

No

The test set should be handled independently of the training set so you could do a separate CV block for the test set if you really wish, and may provide some useful insight but is not universal practice. CV may be useful if you plan to apply the model to a completely new set of ‘real world’ data. Given that the test set is drawn from the same population as the training set this may not be that useful as you would expect it to have similar charcateristics to the training set if the split was performed correctly and without bias. Mind you, may be worth checking this assumption.

What is the purpose of CV?

So long as the aim of performing cross-validation is to acquire a more robust estimate of the test MSE

This is not the purpose of CV, rather it is to estimate the robustness of your performance metrics. As @user86895 states it does not measure MSE, see Mean squared error versus Least squared error, which one to compare datasets? for further reading. CV creates multiple models on subsets of the data and applies them to the data withheld from that subset. It iterates over the dataset, building new models until all have been included in training subsets and all have been included in test subsets. The final model is built on all the training set not any of the individual CV round models, the purpose of CV is not to build models but to assess stability of the model performance, i.e. how generalisable the model is.

When comparing different data processing or analysis algorithms on a dataset it provides a first filter to identify the work pathways that provide the most stable models. It does this by providing estimates of how variable the performance is between sub-sets of your training set. This allows you to detect models with a very high risk of overfitting and filter them out. Without cross validation you would be picking based solely on the maximum performance without concern to its stability. But when you come to apply a model in a deployed situation its stability (relevance across the real world population) will be more important than moderate differences in raw performance on a subset of curated samples (i.e you original experimental set).

Cross validation is in fact essential for choosing the crudest parameters for a model such as number of components in PCA or PLS using the Q2 statistic (which is R2 but on the held out data, see What is the Q² value for each component of a PCA) to determine when overfitting starts to degrade model performance.

If I am mistaken, how could I use the cross-validation result to predict out of sample observations?

I am taking this to mean 'how can I use CV result to estimate performance beyond my experimental set?', but will update this section of my answer if it is clarified differently.

CV is used as a first line estimate of model stability, not to estimate performance in real world settings. The only way to do this is to test the final model in a real-world situation. What CV does is provide you a risk analysis, if it appears stable then you could decide it is time to risk the model on a real-world test. If it is not stable then you need to probably expand your training set considerably (ensuring an even representation of important sub groups and confounding factors as these are one source, other than random noise, of overfitting as all relevant variation needs to be given an equal exposure to the model building process to be properly weighted for) and build a new model.

And a note on real world validation, if it works it doesn’t prove your model is generalisable, only that it works under the specific mechnisms whereby it has been deployed in the real-world.

Answer 2

So long as the aim of performing cross-validation is to acquire a more robust estimate of the test MSE, and not to optimize some tuning parameter, my understanding is that you should use the entire data set.

Correct.

The reason for this is that you would not acquire a model that you can use to predict on unseen test observations, just a measure of MSE for the train set where cross-validation is performed.

Actually you acquire a measure of MSE on test data, not train set

If I am mistaken, how could I use the cross-validation result to predict out of sample observations?

Update:

CV as an evaluation technique provides one output: an evaluation. It does not provide a model that you can use.

CV as a tuning technique provides one output: a model (or component of a model such as the best hyperparameter). It does not provide an evaluation. Hence, the evaluation must be made on an independent test set, which is why data is held out.

In this question "cross-validation to evaluate a statistical learning model", CV I think is intended for evaluation (right?), not tuning. So, you would not use the 10 models from the folds in any way; they are not real models, they are simply byproducts of the evaluative process.

That probably does not answer your question, however. I am guessing you really wanted to know: so, now I have an evaluation, but now I get some new data and want to make predictions. What do I use to make predictions? In other words, you have an evaluation, but not a model. The answer is that you must re-train on the entire dataset. It is this model to which your CV evaluation corresponds.

Answer 3

If I have a data set of 500 observations, should i divide it in a train and test set with for example 375 (75%) train observations and 125 (25%) test observations, and perform cross-validation on the train set?

Yes, you should do it as the initial step, regardless of the situation (test_size=0.2 is also a reasonable default in sklearn).

Or should I perform the cross-validation on the entire data set?

Try to avoid it as much as possible and to stick to the traditional guidelines, but what most guides and books don't tell you is that YOU MAY HAVE TO.
There is theory, and then there is practice... If the labeled data set is small, if there is no practical way to increase it and there is a high risk of the training set being non-representative to a large extent, you may have to give up a separate test set.

Before you launch into non-standard analytic pathways, however, you need to ensure that you understand what the risks associated with them are and the impact on your conclusions. Ultimately, the decision is a pragmatic one based on the needs and the risks associated with the project. In regulated fields, such as finance or healthcare, you need to be cautious, but if all the outcome is whether to colour a 'Buy' button green or blue then the risks are lower.

What you will lose by following this non-standard analytic pathway:

1. you will be giving up the second round of "generalization testing" (with CV on the training set for model selection/tuning being the first one)
2. you will be exposing yourself to the risk of overfitting your data by having no hold out to check whether the model is stable or not when applied to new data
3. you will be performing more hypothesis tests, leading to a higher risk of making a false discovery (related to point 2)
4. you will exploit more researcher degrees of freedom (garden of forking paths) - another route to overfitting

What you will gain:

1. you will be gaining the confidence that your model is trained on the more representative samples
2. you will have a deeper insight into your dataset, hopefully including biases, errors, and the desired data generating processes
3. you will be better placed to recommend improved plans, should any opportunity arise to redo the whole thing

So, with the understanding of the above risks and benefits, how does one make a decision on giving up a separate test set?
Here's a suggested workflow:

1) After selecting and tuning an algorithm using the standard method (training CV + fit on the entire training set + testing on the separate test set), go back to the train/test split, split the data set differently a few times (e.g. using a different random_state parameter value in scikit-learn), each time re-executing the "training CV + train set fit + testing" cycle and observing the scores.

2) If the difference between the training CV scores and the testing score is relatively consistent - great! No need to do anything else.

3) If, however, the difference between the mean training CV score and the testing score is inconsistent, and especially if the fact which one is larger varies between these random data set splitting iterations - YOU'VE GOT A SAMPLING PROBLEM.
Potential solutions:

a) Find another algorithm that will generalize well to the testing set regardless of the train/test split (this may be very challenging, given that you're probably working with a small labeled data set to begin with).

b) Identify and eliminate the splitting bias systematically (this may be not practical either).

c) Keep acquiring more representative labeled data until this problem goes away.

d) Declare this being a "failed analysis" and postpone/cancel model launch, since getting rid of the separate round of generalization testing has high analytic risks associated with it.

e) Only if all else fails, give up a separate test data set (i.e. cut out the separate generalization testing step from the process) and select the model via the cross-validation on the entire labeled data set.
Observe not just the mean CV test score (which is the mean of the fold means) when making model selection/tuning decisions, but also min/max/std of the mean test scores for the individuals folds.