1
$\begingroup$

Let’s consider the following classic model selection + performance estimation strategy for some supervised learning task.

  1. We split our data into train and test dataset in some proportion (e.g., 25% / 75%).
  2. We use train dataset for model selection (algorithm selection, feature selection, hyperparameters tuning) by performing some resampling procedure (e.g., cross-validation) on it.
  3. After we choose final model configuration, we fit it on entire train set.
  4. Using fitted model from step 3, we make predictions on test set, calculate a metric on the test set and use it as an unbiased estimation of model pipeline test error performance.
  5. If everything is ok on step 4, we refit our model configuration on entire dataset and deploy it in prod.

I have a question regarding this strategy: is it more reasonable / valuable to take final model configuration from step 3 and use Cross-Validation on step 4 (split test set into k folds, train model with predefined on step 3 features and hyper parameters on (k-1), predict on last, and so on) instead of just making 1 test prediction? Doing this we get more confident estimations of model pipeline performance (mean, std) instead of 1 number. Let assume that we have enough data in the test set to perform cv on it.

$\endgroup$
3
  • $\begingroup$ Your Q1 is not really cross-validation, but splitting the test set. You would get more estimates of model pipeline performance, but that would not make them more confident: for example the mean of means should be the same as the mean of the whole test set, and splitting them before recombining them brings no benefit. An alternative might be bootstrapping the test set: it will not help with the means (and might not with standard error) but might help with some more complicated statistics. $\endgroup$
    – Henry
    Sep 21 at 23:44
  • $\begingroup$ Your Q2 is really asking why you bother with the test step at all. Just doing cross-validation to estimate performance risks overfitting, as you used performance in cross-validation to make choices in your step 2, and so the data used for that is not the out-of-sample data that you hope your model performs well against. $\endgroup$
    – Henry
    Sep 21 at 23:48
  • $\begingroup$ @Henry I dont understand your point about "Your Q1 is not really cross-validation...". Why? Can use model configuration (algo, hyperparams, features) from step 3, split test set on 5 folds, fit on 1-4, predict on 5 and so on? What makes this procedure different from cv? I will very appreciate some clarification. $\endgroup$
    – kissmemiau
    Sep 22 at 17:30

1 Answer 1

2
$\begingroup$

Split sample validation requires enormous sample sizes to be stable (e.g. n=20,000 for binary Y) and is wasteful of data when developing the model. Use 300-400 bootstrap repetitions instead, or 100 repeats of 10-fold cross-validation. With either bootstrap or CV all analytical steps that may relate to overfitting must be repeated in the inner loop. For large N, you can get away with 10-50 repeats of 10-fold CV.

If you are doing hyperparameter tuning to select the model, that tuning must be repeated inside the inner loop. This may involve turning each inner loop execution into a two-phase process. But note that hyperparameter tuning can be noisy and unreliable with smaller N.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.