2
$\begingroup$

I am a bit confused concerning how I should perform cross-validation to evaluate a statistical learning model.

If I have a data set of 500 observations, should i divide it in a train and test set with for example 375 (75%) train observations and 125 (25%) test observations, and perform cross-validation on the train set? Or should I perform the cross-validation on the entire data set?

So long as the aim of performing cross-validation is to acquire a more robust estimate of the test MSE, and not to optimize some tuning parameter, my understanding is that you should use the entire data set. The reason for this is that you would not acquire a model that you can use to predict on unseen test observations, just a measure of MSE for the train set where cross-validation is performed.

If I am mistaken, how could I use the cross-validation result to predict out of sample observations?

Could someone be kind and clarify this for me?

If relevant, the problem I am solving is performing cross-validation to assess the model performance of a random forest model in R. Thanks in advance!

$\endgroup$
  • $\begingroup$ What do you mean by 'If I am mistaken, how could I use the cross-validation result to predict out of sample observations?' I'm afraid I've read this a couple of times and can't be sure what you mean. Do you mean 'how can I use CV result to estimate performance beyond my experimental set?' $\endgroup$ – ReneBt Apr 11 '18 at 8:58
3
$\begingroup$

should i divide it in a train and test set with for example 375 (75%) train observations and 125 (25%) test observations, and perform cross-validation on the train set?

Yes

Or should I perform the cross-validation on the entire data set?

No

The test set should be handled independently of the training set so you could do a separate CV block for the test set if you really wish, and may provide some useful insight but is not universal practice. CV may be useful if you plan to apply the model to a completely new set of ‘real world’ data. Given that the test set is drawn from the same population as the training set this may not be that useful as you would expect it to have similar charcateristics to the training set if the split was performed correctly and without bias. Mind you, may be worth checking this assumption.

What is the purpose of CV?

So long as the aim of performing cross-validation is to acquire a more robust estimate of the test MSE

This is not the purpose of CV, rather it is to estimate the robustness of your performance metrics. As @user86895 states it does not measure MSE, see Mean squared error versus Least squared error, which one to compare datasets? for further reading. CV creates multiple models on subsets of the data and applies them to the data withheld from that subset. It iterates over the dataset, building new models until all have been included in training subsets and all have been included in test subsets. The final model is built on all the training set not any of the individual CV round models, the purpose of CV is not to build models but to assess stability of the model performance, i.e. how generalisable the model is.

When comparing different data processing or analysis algorithms on a dataset it provides a first filter to identify the work pathways that provide the most stable models. It does this by providing estimates of how variable the performance is between sub-sets of your training set. This allows you to detect models with a very high risk of overfitting and filter them out. Without cross validation you would be picking based solely on the maximum performance without concern to its stability. But when you come to apply a model in a deployed situation its stability (relevance across the real world population) will be more important than moderate differences in raw performance on a subset of curated samples (i.e you original experimental set).

Cross validation is in fact essential for choosing the crudest parameters for a model such as number of components in PCA or PLS using the Q2 statistic (which is R2 but on the held out data, see What is the Q² value for each component of a PCA) to determine when overfitting starts to degrade model performance.

If I am mistaken, how could I use the cross-validation result to predict out of sample observations?

I am taking this to mean 'how can I use CV result to estimate performance beyond my experimental set?', but will update this section of my answer if it is clarified differently.

CV is used as a first line estimate of model stability, not to estimate performance in real world settings. The only way to do this is to test the final model in a real-world situation. What CV does is provide you a risk analysis, if it appears stable then you could decide it is time to risk the model on a real-world test. If it is not stable then you need to probably expand your training set considerably (ensuring an even representation of important sub groups and confounding factors as these are one source, other than random noise, of overfitting as all relevant variation needs to be given an equal exposure to the model building process to be properly weighted for) and build a new model.

And a note on real world validation, if it works it doesn’t prove your model is generalisable, only that it works under the specific mechnisms whereby it has been deployed in the real-world.

$\endgroup$
3
$\begingroup$

So long as the aim of performing cross-validation is to acquire a more robust estimate of the test MSE, and not to optimize some tuning parameter, my understanding is that you should use the entire data set.

Correct.

The reason for this is that you would not acquire a model that you can use to predict on unseen test observations, just a measure of MSE for the train set where cross-validation is performed.

Actually you acquire a measure of MSE on test data, not train set

If I am mistaken, how could I use the cross-validation result to predict out of sample observations?

Update:

CV as an evaluation technique provides one output: an evaluation. It does not provide a model that you can use.

CV as a tuning technique provides one output: a model (or component of a model such as the best hyperparameter). It does not provide an evaluation. Hence, the evaluation must be made on an independent test set, which is why data is held out.

In this question "cross-validation to evaluate a statistical learning model", CV I think is intended for evaluation (right?), not tuning. So, you would not use the 10 models from the folds in any way; they are not real models, they are simply byproducts of the evaluative process.

That probably does not answer your question, however. I am guessing you really wanted to know: so, now I have an evaluation, but now I get some new data and want to make predictions. What do I use to make predictions? In other words, you have an evaluation, but not a model. The answer is that you must re-train on the entire dataset. It is this model to which your CV evaluation corresponds.

$\endgroup$
  • $\begingroup$ Thanks for answering! To claify: If I were to somehow use the results from the cross-validation on new data, how could you do that? Because the 10 fits resulting from a 10 fold cv for instance are very different, and you are not left with a "cv model" if i understand correctly. The reason im asking, is because i often see that people perform cv on a training set, while additionally have a held out test set. Im confused about what the purpose of this held out test set then is. $\endgroup$ – Kaangy Apr 10 '18 at 19:25
  • $\begingroup$ That makes sense. "CV as a tuning technique provides one output: a model. It does not provide an evaluation. Hence, the evaluation must be made on an independent test set, which is why data is held out." Doesn't the CV as a tuning technique provide only the optimal tuning value, e.g. number of K neighbors, which you then will apply to the training set, but not a model in itself? $\endgroup$ – Kaangy Apr 10 '18 at 19:48
  • $\begingroup$ Technically yes it provides the performance of the model under k. Then you should take the best k and retrain on the full training set. $\endgroup$ – user0 Apr 10 '18 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.