As I understand it, performing nested CV means that you have two layers of CV, and you use each layer for one and only one thing:
- The outer layer, for estimating the quality of the models trained on the inner layer.
- The inner layer for selecting the best model (including parameters, hyperparameters and so on).
One important distinction: in fact, you are not assessing the quality of the model, but of the procedure of model selection. What you are evaluating in the outer folds is that the procedure for model selection is consistent, and the quality of the estimators built on the outer loops
So, for each k-outer-fold, you will select one (and only one) inner model (the best one), trained on the training set for this k-outer-fold, and this model will be evaluated on the validation set for this k-outer-fold. After you vary the outer k, you will have k-outer estimates and you can average them to assess better the quality of the models.
Another important thing: once you find the best hyperparameters for a particular loop, how can you select the model, if you have k-inner-folds models? You can do two things: create a new instance of the model with this parameters and fit it on the whole inner fold, or create an ensemble of all k-inner-fold models. Both can be ok.
I have released a package that can help implementing nested cross validation in Python (for the moment, it only works for binary classifiers). If you want to check it out, it's here:
https://github.com/JaimeArboleda/nestedcvtraining
It's my first Python package, so any comments, suggestions or critics will be more than welcome!!
I post it as an answer because nested cross validation is performed inside the main function and you don't have to take care of how to implement it. Anyway, the code is visible in my github account, so that you can check how I implemented it if you are curious. Also, the readme explain it in detail (although, maybe, not very clearly!).
After the function is called, you get a model (that in fact will be a pipeline if there is a post-process to perform) and a complete report of what happened inside, so that you can assess the estimates of the different scoring functions. It comes with many options thay may be enough for a lot of common settings, I hope.
I don't test with a completely separated test set, because the outer procedure is meant to do this repeatedly and you can assess the quality of the model. However, if you have any doubt I'll be more than happy to try to answer it.