Let’s consider the following classic model selection + performance estimation strategy for some supervised learning task.
- We split our data into train and test dataset in some proportion (e.g., 25% / 75%).
- We use train dataset for model selection (algorithm selection, feature selection, hyperparameters tuning) by performing some resampling procedure (e.g., cross-validation) on it.
- After we choose final model configuration, we fit it on entire train set.
- Using fitted model from step 3, we make predictions on test set, calculate a metric on the test set and use it as an unbiased estimation of model pipeline test error performance.
- If everything is ok on step 4, we refit our model configuration on entire dataset and deploy it in prod.
I have a question regarding this strategy: is it more reasonable / valuable to take final model configuration from step 3 and use Cross-Validation on step 4 (split test set into k folds, train model with predefined on step 3 features and hyper parameters on (k-1), predict on last, and so on) instead of just making 1 test prediction? Doing this we get more confident estimations of model pipeline performance (mean, std) instead of 1 number. Let assume that we have enough data in the test set to perform cv on it.