With nested CV, how can we get a sense of how our model may perform in the outer loop before actually bringing it to the outer loop?
Without nested CV, if I did simple 10-Fold CV on training data (80%) for model selection, I would be able to get a good sense of which models are performing best and which to apply to held out test set (20%) for generalization error estimation. I may also get a good sense from the training data with this approach of whether I need to go back to the drawing board and think about potentially including additional variables, trying different models, or pre-processing the data differently, before applying any models to the held out test data.
However, with nested CV, this is less intuitive to me: If for example my full dataset has 1000 patients, with 10-fold nested CV, one thought would be to take a single inner loop of 900 patients and evaluate how well the models are doing on this inner loop without looking at the single held out outer loop. Is this reasonable? However, my concerns with this approach are (a) Any decisions to modify the models (e.g., adding or transforming a predictor) based on this inner loop of 900 patients would lead to bias because the same patients in this inner loop of 900 patients are the same patients in the outer loop of a different fold which in my eyes is the equivalent therefore of making decisions based on results of the test set. Or can we for that matter look at the results of the models from each of the 10 inner loops and then go back to the drawing board if performance seems poor on each of these inner loops - or is using just one of the inner loops sufficient to understand whether model adjustments may be necessary? (b) I am using only a single fold and would potentially make different decisions on model adjustment if I considered the other 9 folds - although, as I understand it, using a different fold should ideally in theory lead to the same decision.