In k-fold cross validation does the training subsample include test set?

In this Wikipedia page in subsection for K-fold cross validation it says "In k-fold cross-validation, the original sample is randomly partitioned into k equal size subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data". Test data is not in the picture at all.

Whereas in a book I read the author clearly indicates

1. The full data is divided into three sets: Training set, test set, and validation set (or subsamples in Wikipedias language).
2. Of the k subsamples one subsample is retained as the validation data, one other subsample is retained as the test data, and k-2 subsamples are used as training data.

Which is true?

• Side note: "validation" has different meanings in different fields. Some authors use it to indicate a kind of testing that is used to select models (I'd suggest that optimization test set would be more intuitive), whereas in other fields validation means demonstrating that the final model is fit for its purpose. Mar 17, 2014 at 16:48

They are both correct in their own context. They are describing two different ways of model selection in different situations.

In general, when you are doing model selection and testing, your data is divided into three parts, training set, validation set and testing set. You use your training set to train different models, estimate the performance on your validation set, then select the model with optimal performance and test it on your testing set.

On the other hand, if you are using K-fold cross-validation to estimate the performance of a model, your data is then divided into K folds, you loop through the K folds and each time use one fold as testing(or validation) set and use the rest (K-1) folds as training set. Then you average across all folds to get the estimated testing performance of your model. This is what the Wikipedia page is referring to.

But keep in mind that this is for testing a specific model, if you have multiple candidate models and want to do model-selection as well, you have to select a model only with your training set to avoid this subtle circular logic fallacy. So you further divide your (K-1) folds 'training data' into two parts, one for training and one for validation. This means you do an extra 'cross-validation' first to select the optimal model within the (K-1) folds, and then you test this optimal model on your testing fold. In other words, you are doing a two-level cross-validation, one is the K-fold cross-validation in general, and within each cross-validation loop, there is an extra (K-1)-fold cross-validation for model selection. Then you have what you stated in your question, 'Of the k subsamples one subsample is retained as the validation data, one other subsample is retained as the test data, and k-2 subsamples are used as training data.'

• The two-level cross validation is also known as double or nested cross validation. While nested CV and the split-into-3-sets strategies are similar in that both use one test set for selection/optimization and the other for assessing the optimized model's performance, they are not the same: nested k-fold cross validation builds k * k' (usually = k - 1) * no. of tested hyperparameter combination surrogate models, the direct 3-set strategy only one for each hyperparameter combination. Mar 17, 2014 at 16:54
• (The Wiki quote does not speak about the purpose of the validation at all which may be different from selection.) Mar 17, 2014 at 16:54
• can you confirm my pseudocodes?
– ozi
Mar 24, 2014 at 22:19

Here I am re-stating what I gathered from the answer of @Yuanning and comments of @cbeleites in pseudocode form. This may be helpful for people like me.

To measure perfomance of a determined model we need only training and test sets:

function measure_performance(model, full_test_set, k_performance):
subset_list <- divide full_test_set into k_performance subsets
performances <- empty array
for each sub_set in subset_list:
test_set <- sub_set
training_set <- the rest of the full_test_set
model <- train model with training_set
performance <- test model with test_set
append performance to performances
end for each
return mean of the values in peformances
end function


But if we need to do model selection, we should do this:

function select_model(data, k_select, k_performance):
subset_list <- divide data into k_select subsets
performances <- empty array
for each sub_set in subset_list:
validation_set <- assume that this sub_set is validation set
test_set <- one other random sub_set (Question: How to select test_set)
training_set <- assume remaining as training set
model <- get a model with the help of training_set and validation_set
performance <- measure_performance(model,test_set, k_performance)
end for each
return model with the best performance (for this, performances will be scanned)
end function

• Is k_performance intended to be the label for a specific fold among the total number k_select performance folds? I suppose this goes to your question about "How to select test_set". I would also appreciate specific feedback on this. I have seen this 'nested' cross-validation for model selection alluded to, but never described in detail. Mar 21, 2017 at 19:07
• In the context of cross-validation, the problem of model selection based only on the training data (even if cross-validated within that data) is specifically articulated here: en.wikipedia.org/wiki/…. The assertion (based on several references) is that the cross-validated performance estimates based wholly on the training data are subject to high variance (i.e., with respect to the choice of training data), in spite of the cross-validation procedure. Mar 21, 2017 at 19:30