Do I need an initial train/test split for nested cross-validation?

Question

I have a couple of pipelines:

• pipeline 1: CV'd feature selection, CV'd hyperparameter selection for classifier A
• pipeline 2: CV'd feature selection, CV'd hyperparameter selection for classifier B
• pipeline 3: CV'd feature selection, CV'd hyperparameter selection for classifier C
• pipeline 4: CV'd feature selection, CV'd hyperparameter selection for classifier D

I want to figure out what the best model process is. So I put all of this into another CV loop to do nested CV:

for pipeline in [pipeline1, pipeline2, pipeline3, pipeline4]:
    for folds in my CV:
        run pipeline
        score pipeline
    get average score across folds for pipeline

That should give me an average score for each pipeline, and I choose the one that maximizes my score.

But if I want a final unbiased estimate of model performance, do I:

1. Use the average score from the CV loop?
2. Split data into a train/test BEFORE I run the nested-CV, and then run nested-CV on train, choose my model, and get a final performance metric from training it on the initial train set and testing it on the test set?

Answer 1

Using the first approach of the average score of your CV loop would give overly optimistic accuracies because when you selected the hyper-parameters, you chose the ones which gave you the highest accuracy. Effectively, you would have no out of sample testing using that method. It would be like trying to find the highest score of a basketball game in a season by summing up the number of points each player got in the game where they individually scored the most points (where the game would be the hyper-parameters and the players the folds of cross-validation).

The most standard way to do the evaluation would be your second suggestion where you do cross-validation on a training set and then do the evaluation on a held out test set. This ensures that the model selection is independent of your testing data giving you a more realistic metric for the generalization of your model.

One final method I have seen, particularly in small samples where data is limited, is to use a nested cross-validation. In this version, your data are divided up into N groups. In the first fold, group 1 would be the testing group and groups 2-N would be the training groups. Cross-validation would be performed on groups 2-N to select model hyper-parameters. The model is then trained on that training set (groups 2-N) and tested on group 1. In the next fold group 2 is held out and groups 1,3-N are used to train the model. This continues for the total of N-folds.

While it is clear to see the advantage of this method in small data (you can use a large amount of your data to train the models and still have all of the data available for evaluation) it can cause problems in later evaluation. First off, your results now come from N different models so the errors can't easily be compared (what if there was just one fold that chose really bad parameters but the rest were great). Secondly, while the results come from different models, those models are not independent as (N-2)/(N-1) of the training data were the same between any two pairs of models. Most statistical tests require an assumption of independence so that can lead to difficulty selection a metric for evaluation.

In general, I would go with your approach 2, but if data is really limiting, it may be worth doing nested cross-validation. I cannot imagine a circumstance where the first approach would be appropriate.