When using nested cross-validation, can different models be explored in a principled way, without spoiling the final estimate of model performance?
When evaluating a model, the data used to build or train the model should not be used to estimate its performance on future data, as its performance estimate would be upwardly biased. This leads to a split of an initial dataset into training and testing. In order to acquire a measure of performance variance, and to include all data in testing, a form of cross-validation is often used, where the initial dataset is split into several training and test sets, with final model performance collected across the test sets.
As described elsewhere, e.g. When is nested cross-validation really needed and can make a practical difference?, if model selection is performed, including the selection of model hyper-parameters, this needs to be implemented via an inner cross-validation within each training fold, leading to nested cross-validation.
Appendix B of the book “Hands-On Machine Learning with Scikit-Learn and TensorFlow” by Aurélien Géron describes, for a situation in which the final test data is completely discrete from the training and validation data, exploring a dataset by fitting multiple models:
Short-List Promising Models
If the data is huge, you may want to sample smaller training sets so you can train many different models in a reasonable time (be aware that this penalizes complex models such as large neural nets or Random Forests).
Once again, try to automate these steps as much as possible.
Train many quick and dirty models from different categories (e.g., linear, naive Bayes, SVM, Random Forests, neural net, etc.) using standard parameters.
Measure and compare their performance.
For each model, use N-fold cross-validation and compute the mean and standard deviation of the performance measure on the N folds.
Analyze the most significant variables for each algorithm.
Analyze the types of errors the models make.
What data would a human have used to avoid these errors?
Have a quick round of feature selection and engineering.
Have one or two more quick iterations of the five previous steps.
Short-list the top three to five most promising models, preferring models that make different types of errors.
This is all sound because the final model will be evaulated on a separate, held-out dataset, again from Géron, Appendix B:
Don’t tweak your model after measuring the generalization error: you would just start overfitting the test set.
My specific question is whether similar exploratory data analysis procedures can be followed when using nested cross-validation for final model performance estimation, without biasing your performance estimate.
My first intuition was that it wouldn't be possible to explore the data in the same way when using nested cross-validation for your final estimate, since you don't have a discrete test data set after the tuning has been performed. However on further thought it seems that the approach described by Géron is essentially performing the inner model-selection loop of the nested cross-validation manually, so perhaps there is a procedure in which it would be performed in a principled way, for example by setting up a nested cross-validation structure and only viewing the inner loop CV performance of the various models explored.