Is it reasonable to perform cross-validation on the test dataset?

Question

Let’s consider the following classic model selection + performance estimation strategy for some supervised learning task.

1. We split our data into train and test dataset in some proportion (e.g., 25% / 75%).
2. We use train dataset for model selection (algorithm selection, feature selection, hyperparameters tuning) by performing some resampling procedure (e.g., cross-validation) on it.
3. After we choose final model configuration, we fit it on entire train set.
4. Using fitted model from step 3, we make predictions on test set, calculate a metric on the test set and use it as an unbiased estimation of model pipeline test error performance.
5. If everything is ok on step 4, we refit our model configuration on entire dataset and deploy it in prod.

I have a question regarding this strategy: is it more reasonable / valuable to take final model configuration from step 3 and use Cross-Validation on step 4 (split test set into k folds, train model with predefined on step 3 features and hyper parameters on (k-1), predict on last, and so on) instead of just making 1 test prediction? Doing this we get more confident estimations of model pipeline performance (mean, std) instead of 1 number. Let assume that we have enough data in the test set to perform cv on it.

Answer 1

Split sample validation requires enormous sample sizes to be stable (e.g. n=20,000 for binary Y) and is wasteful of data when developing the model. Use 300-400 bootstrap repetitions instead, or 100 repeats of 10-fold cross-validation. With either bootstrap or CV all analytical steps that may relate to overfitting must be repeated in the inner loop. For large N, you can get away with 10-50 repeats of 10-fold CV.

If you are doing hyperparameter tuning to select the model, that tuning must be repeated inside the inner loop. This may involve turning each inner loop execution into a two-phase process. But note that hyperparameter tuning can be noisy and unreliable with smaller N.