Nested Cross Validation - How to Improve Models Without Bias

Question

With nested CV, how can we get a sense of how our model may perform in the outer loop before actually bringing it to the outer loop?

Without nested CV, if I did simple 10-Fold CV on training data (80%) for model selection, I would be able to get a good sense of which models are performing best and which to apply to held out test set (20%) for generalization error estimation. I may also get a good sense from the training data with this approach of whether I need to go back to the drawing board and think about potentially including additional variables, trying different models, or pre-processing the data differently, before applying any models to the held out test data.

However, with nested CV, this is less intuitive to me: If for example my full dataset has 1000 patients, with 10-fold nested CV, one thought would be to take a single inner loop of 900 patients and evaluate how well the models are doing on this inner loop without looking at the single held out outer loop. Is this reasonable? However, my concerns with this approach are (a) Any decisions to modify the models (e.g., adding or transforming a predictor) based on this inner loop of 900 patients would lead to bias because the same patients in this inner loop of 900 patients are the same patients in the outer loop of a different fold which in my eyes is the equivalent therefore of making decisions based on results of the test set. Or can we for that matter look at the results of the models from each of the 10 inner loops and then go back to the drawing board if performance seems poor on each of these inner loops - or is using just one of the inner loops sufficient to understand whether model adjustments may be necessary? (b) I am using only a single fold and would potentially make different decisions on model adjustment if I considered the other 9 folds - although, as I understand it, using a different fold should ideally in theory lead to the same decision.

Answer 1

how can we get a sense of how our model may perform in the outer loop before actually bringing it to the outer loop?

If there wasn't any hyperparameter tuning using the inner CV results, they'd be a very good estimate for outer loop test results (just slightly pessimistically biased due to having slightly less training data).

The optimistic bias is caused by selecting based on uncertain performance estimates: that has the risk of selecting a model/hyperparameter set that just accidentally looked better than another one. Thus, the expected optimistic bias is

• the lower the fewer models you compare
• the lower the more [inner test] cases you have
• the lower the more stable the compared models are

You may say that high optimistic bias inner CV vs. outer CV means that your data set did not contain sufficient information for the optimization.

---

Any decisions to modify the models (e.g., adding or transforming a predictor) based on this inner loop of 900 patients would lead to bias

yes (but that doesn't hurt: you still have the outer held out cases to get an unbiased performance estimate. That's why you need the outer loop)

 because the same patients in this inner loop of 900 patients are the same patients in the outer loop of a different fold which in my eyes is the equivalent therefore of making decisions based on results of the test set

no, that's not the reason for the optimistic bias: see above

The folds of the outer loop are independent of each other in the sense that for each of the folds, a completely fresh inner optimization is run. No information gets from outer-fold 1's inner loop to outer fold 2's inner loop (i.e. is 10-fold inner nested inside 10 fold outer CV, a total of 110 or 121 models are calculated: 10 x (10 for hyperparameter tuning + 1 optimized) during the cross validation and then the final model with 10 inner folds to determine hyperparameters and finally with these hyperparameters on the whole data set the final model).
OTOH, the overlap in training sets is intended: we implicitly assume that the 10 folds of a cross validation arrive at (almost) the same model as they are trained on almost the same training data.

Or can we for that matter look at the results of the models from each of the 10 inner loops and then go back to the drawing board if performance seems poor on each of these inner loops

Yes, that's perfectly fine. It will create a bias in the sense that the inner CV performance estimates may be too optimistic, but again that's why you reserved the outer test data.

As you don't want to manually do all these decisions 100 times (10 outer x 10 inner), you'll want to code your heuristic for automatic evaluation once it has evolved from your manual supervision.

The only thing you're not allowed to do are sneak previews on the outer test data.

---

(b) I am using only a single fold and would potentially make different decisions on model adjustment if I considered the other 9 folds - although, as I understand it, using a different fold should ideally in theory lead to the same decision.

see answer to your other question. You'll want to decide whether you observe harmful variation i.e. your optimization doesn't work or harmless variation (e.g. some models are invariant to certain transformations)

I'm frequently in a situation where I know that lots of changes in the model parameters may be irrelevant (e.g. PCA flipping axes). In your case, you could e.g. obtain the same boundary in sample space for LR and linear SVM.
In such situations it may help to look at the stability of the predictions instead (for that, you can replace the single run of CV [for outer and/or inner CV] by repeated aka iterated cross validation).