Do the selected hyperparameters from the inner loop of a nested k-fold cross validation still induce bias?

Question

I think the nested k-fold cross validation does not really provide an unbiased estimate of performance.

If I understand correctly, in the image below, we will first perform a 10-fold cross validation with each training dataset on the right (inner loop), performing hyperparameter tuning.

I believe that the result is biased if we pick the best model from the inner loop by averaging the performance over the different inner loop training set K-folds (all the orange/red rows on the right) and then retraining and test on the outer loop test set.

I think that is because each yellow box on the each row on the left (outer loop) might not be part of the corresponding row on the right (inner loop), but it has essentially been used for hyperparameter tuning by calculating the K-fold CV score of the inner loop for the rest of the datasets.

For example, the test data (yellow box) of the first row in the outer loop has been used as training data by being incorporated in all the other training sets in the inner loop (2nd, 3rd, 4th, 5th).

Is my logic flawed or have I misunderstood how nested cross validation works?

Answer 1

That's interesting! Let's assume for having a more simplified approach that k_inner_folds = k_outer_folds = 3. Let's call the partitions: A, B, C for the outer loop and AB1, AB2, AB3, AC1, AC2, AC3, BC1, BC2, BC3 for the inner loop respectively. Ok, we have three outer loops:

• First one: AB as training set for the inner loop. C for test set. In this loop, you will try lots of combinations of parameters. For each combination, P, you will:
  • Train three models, one using AB1-AB2 as training set, that you will test on AB3; one using AB2-AB3 as training set, that you will test on AB1; one using AB1-AB3 as training set, that you will test on AB2.
  • You will average the scores in order to have a measure for the quality of the parameters P.
  • By using this approach, you will have a mapping P -> Score, that will let you select the best parameters Pbest (the combination that maximizes the score).
  • In all of this, you have not used at all C. Then, need an unique model, and you have two options: creating an ensemble of the three models of the Pbest or fitting a new instance of model with Pbest parameters on the whole AB dataset. In any case, this model will be checked against C, that is completely new. As you see, the first loop is completely safe from the point of view of the bias of the estimation.
• Second one... The same as before. You will get another safe estimation of the quality of the model.
• Third one... The same.

At the end, you average the three unbiased scores and will end up with an unbiased (and sharper, with less variance) estimation of the quality of your model. In fact, you are assessing more than the quality of the model: you are assessing the quality of your procedure for model selection, so after you do that, if the scores are good, you can safely repeat the inner procedure with the whole initial dataset ABC. And your estimation will in fact be a pessimistic one, because you now have more data, and if the learning rate is still positive, you can build a little better model now.

If you are interested, I have created a package that does what I have described an have some features that might come in handy in some settings. I did it for my own projects, but I hope someone finds it useful. It's here:

https://github.com/JaimeArboleda/nestedcvtraining