Is resampling to obtain a test set ok?

Question

I have a question about cross-validation and hyperparameter optimization. Namely about how to test the final model performance in an unbiased way.

Now, I know I have to train the model, optimize the hyperparameters and test the model on separate datasets. But I'm wondering if I can first optimize the hyperparameters on some validation data, then resample my dataset to create new train and test sets and evaluate the model with the chosen hyperparameters? Specifically, I have a small dataset and a neural net model which is proving difficult to train and optimize. The models performance appears to vary quite a bit based on the train-test splits, so I want to use 5-fold cross validation. The procedure I'm thinking of doing is:

1. Do 5-fold cross validation on the entire dataset, to optimize the hyperparameters.
2. Split the dataset into a new train and test set
3. Train a model on the train set using the best hyperparameters from step 1. Evaluate on the test set.

The question is, will this give unbiased estimates of the error, since the model is new and these exact train-test sets were never used in hyperparameter optimization? Or will the error be underestimated, because the same datapoints were used in original cross-validation, so that the hyperparameter might have overfit to this data?

Answer 1

Unfortunately, this is not ok. If you follow the proposed procedure, the final error estimate (using the test set) will be downwardly biased, and the model will appear to generalize better than it actually will. This is because data from the test set has already been used to select the hyperparameters. To obtain an (asymptotically) unbiased estimate of generalization performance, the test set must be independent from data used to fit the model, including hyperparameter tuning.

If the dataset is small, use nested cross validation. This will properly maintain independence of the training/validation/test data, but yield lower variance error estimates compared to simple holdout (single training/validation/test split). To reduce the variance even further, use repeated nested cross validation (i.e. repeat nested cross validation multiple times, partitioning the data randomly each time, then average the error across repetitions).

Answer 2

Excellent answer from @user20160, there's one more thing to add to the original question:

When selecting models with cross-validation, sample set is usually split into 2 parts: Part 1 is the traning & validation set, part 2 is the test set.

The general steps are:

• Step 1. Split the samples into two parts, part 1 for traning and validation, part 2 for model evaluation (the "test set").

• Step 2. Select the best model (hyperparameter) by performaing cross validation on part 1. Say after this step, the best hyper parameter $\gamma$ is selected. NOTE: part 2 should never be involved in this step.

• Step 3. Train your model with hyper parameter $\gamma$ on the whole part 1. Say after this step, you get the trained model $M$.

• Step 4. Evaluate $M$ with part 2. AND THAT'S IT, $M$ IS YOUR BEST MODEL, NEVER GO BACK TO ANY PREVIOUS STEPS.

In many practical cases you may only want to pick the best model, without wanting to evaluate how good/bad the best model performs, at least not for now.

For example when sample size is small, you want to make full use of it and postpone the model evaluation to a later stage when more samples arrive. In this case you don't need to split the sample sets into two parts, instead the whole sample set palys as the training & validation set ("part 1"), and only step 1-3 is needed.