# Does changing the parameter search space after nested CV introduce optimistic bias?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space to c:[0.5,1,1.5,2,2.5,3,3.5,4].

Can I just repeat nested CV now and expect a fair estimate of generalization error? If not, what to do? It seems that I have changed my parameter space (which is a part of the modeling process) based on knowledge I now have from using the whole dataset in nested CV, and therefore need to evaluate on a dataset external to my current dataset rather than using nested CV on the same dataset. I am not "choosing" parameters explicitly based on the test data, but I am allowing for their choice based on the testing data. Should I perform nested CV on a subset of the data to find good parameter spaces and then repeat on the full data? It seems that doing so would still allow the algorithm to see some of the data while choosing parameters and in worst case that "some" of the data randomly finds its way into every test fold upon repeated nested CV.

For CV to give an unbiased estimate, every step of the modeling process must be repeated independently in each fold. Isn't selection of a search space for hyperparameters a "step" in the modeling process?

• If you just expand the search space for the one fold under consideration, you avoid this problem. Consider a function train_and_tune that does the inner CV, and, if necessary expands search space as needed. train_and_tune works independently for all outer folds. If your tuning is stable, all those independent runs of train_and_tune will result in the same (very similar) tuned models. If those models vary all over the place, you found a problem. But at least the related bad outer performance is an honest estimate of performance (i.e. a proper judgment of a failed optimization) . – cbeleites Jun 19 '17 at 16:17