# Do I need an initial train/test split for nested cross-validation?

I have a couple of pipelines:

• pipeline 1: CV'd feature selection, CV'd hyperparameter selection for classifier A
• pipeline 2: CV'd feature selection, CV'd hyperparameter selection for classifier B
• pipeline 3: CV'd feature selection, CV'd hyperparameter selection for classifier C
• pipeline 4: CV'd feature selection, CV'd hyperparameter selection for classifier D

I want to figure out what the best model process is. So I put all of this into another CV loop to do nested CV:

for pipeline in [pipeline1, pipeline2, pipeline3, pipeline4]:
for folds in my CV:
run pipeline
score pipeline
get average score across folds for pipeline


That should give me an average score for each pipeline, and I choose the one that maximizes my score.

But if I want a final unbiased estimate of model performance, do I:

1. Use the average score from the CV loop?
2. Split data into a train/test BEFORE I run the nested-CV, and then run nested-CV on train, choose my model, and get a final performance metric from training it on the initial train set and testing it on the test set?
• The same CV loop that is used for model selection cannot be used for unbiased performance estimate. This is why one needs nested CV in the first place. And the same logic applies here too. So if you want truly unbiased, you should do (2). Good question, +1. Aug 17 '16 at 23:19