That's interesting! Let's assume for having a more simplified approach that k_inner_folds = k_outer_folds = 3. Let's call the partitions: A, B, C for the outer loop and AB1, AB2, AB3, AC1, AC2, AC3, BC1, BC2, BC3 for the inner loop respectively. Ok, we have three outer loops:
- First one: AB as training set for the inner loop. C for test set. In this loop, you will try lots of combinations of parameters. For each combination, P, you will:
- Train three models, one using AB1-AB2 as training set, that you will test on AB3; one using AB2-AB3 as training set, that you will test on AB1; one using AB1-AB3 as training set, that you will test on AB2.
- You will average the scores in order to have a measure for the quality of the parameters P.
- By using this approach, you will have a mapping P -> Score, that will let you select the best parameters Pbest (the combination that maximizes the score).
- In all of this, you have not used at all C. Then, need an unique model, and you have two options: creating an ensemble of the three models of the Pbest or fitting a new instance of model with Pbest parameters on the whole AB dataset. In any case, this model will be checked against C, that is completely new. As you see, the first loop is completely safe from the point of view of the bias of the estimation.
- Second one... The same as before. You will get another safe estimation of the quality of the model.
- Third one... The same.
At the end, you average the three unbiased scores and will end up with an unbiased (and sharper, with less variance) estimation of the quality of your model. In fact, you are assessing more than the quality of the model: you are assessing the quality of your procedure for model selection, so after you do that, if the scores are good, you can safely repeat the inner procedure with the whole initial dataset ABC. And your estimation will in fact be a pessimistic one, because you now have more data, and if the learning rate is still positive, you can build a little better model now.
If you are interested, I have created a package that does what I have described an have some features that might come in handy in some settings. I did it for my own projects, but I hope someone finds it useful. It's here: