Cross-validation when splitting data into train/dev/test sets

Question

Background:

• Train set: data used to train the chosen model
• Dev set: data used to tune the model's parameters
• Test set: data used to evaluate the performance of the final model

How cross-validation is done when splitting the data into a train set, a dev set and a test set instead of just train/test sets? I could not find any reference on this matter in the litterature.

My intuition would be to perform a two-step cross-validation. For instance, if we want to do a 10 fold cross-validation, we would do a first a basic 10 fold cross validation to separate the train and test set. And then we will split up the train set into train and dev set using a 9 (10-1) cross-validation. We will end up with 80% train, 10% dev, 10% test.

This method respects the generalization wanted by the cross-validation methodology. However the number of computations is (almost) squared, which is huge.

Another possibility would be to do a 5 (10/2) fold cross-validation to split the data into train and dev+test set. And split the dev+test set at the middle to recover the dev and test sets individually. We will also end up with 80% train, 10% dev and 1°% test.

What is your opinion on this ?

Answer 1

I will just answer my own question, since I have found the answer I was looking for.

What I wanted to do is a nested cross-validation. It is made of:

• An inner loop that is doing the model tuning using K-fold cross-validation for every combination parameters (using grid search or random search)
• An outer loop doing the model evaluation using K'-fold cross-validation using the best combination of parameters of the inner loop

So if I want to split my data into 80%/10%/10% (train/dev/test), I will first do a 10-fold cross validation (outer loop: 90% (train+dev) and 10% test). And for every fold, I will do a 9-fold cross-validation to select the best parameters (inner loop: 80% train and 10% dev).

How does it compare to a simple cross-validation in terms of number of trained models ?

• CV: K models instead of 1
• nested-CV: K * K' * number of parameters combinations

So the nested-CV is much more expensive than basic CV and depends on the paramrter space size (if doing grid search) and the CV factor of the inner loop. Smart parameter space restriction could speed up de computations quite a bit.

Additional information on nested-CV:

• Nested cross validation for model selection
• https://www.elderresearch.com/blog/nested-cross-validation

Answer 2

Philosophically speaking,

Cross validation = splitting your data into train and dev

So when you have split your data into two sets (train/dev) and you know that these cannot be used in order to evaluate the model performance (because you need this split for tuning model hyperparameters) then "no cross validation is necessary". I state that in quotes because sometimes it is better to use cross validation instead of just one single split into train/dev in order to get statistically more "stable" results.

However, there are situations where it is not senseful to do cross validation (and one should in fact use a single split into train and test) or at least one should do it carefully: In many of the use cases I see, the data we get has some kind of ordering axis. For example, if we want to predict (for example) the length of of taxi rides then there is a time component: the date on which the ride takes place. There may be some hidden seasonalities (in winter it takes longer because the are more accidents and the streets are more crowded[?]). Suppose we have data ranging from 2018-01-01 until 2018-05-01. I would then train the model on train=data from 2018-01-01 until 2018-03-31 and test the hyperparameter performance on dev=data from 2018-04-01 until 2018-05-01. I would not recommend to do a 'blind' X fold cross validation here because if you did, some data points from may (a summer month) would end up in the training set aloowing your model to 'peek' into the sunny future. Then it could be that you overestimate the performance because when training in the real world in april you only get winter data in order to train the model and then it has to work on summer data.

Of course the model would then perform badly and you would then get more data and train the model on a whole year in order for it so learn these seasonalities but in order to spot this out you deliberately needed to split into train and dev only once (and according to the time axis).