What is the correct procedure for nested cross-validation?

Question

I am trying to use scikit-learn to make a classifier and then predict the accuracy of the classifier. My dataset is relatively small and I am unsure of the best parameters. Hence I turned to nested cross-validation (nCV) to make and test my model.

I have been trying to understand the best methodology. However after reading:

1. Do I need an initial train/test split for nested cross-validation?
2. Cross Validation Vs Train Validation Test
3. How to split the dataset for cross validation, learning curve, and final evaluation?

I am still at a loss as to the best way to proceed.

So far I have:

1. Split (80%/20%) the entire data set into training and testing sets
2. Defined my inner-cv, outer-cv, parameter grid and estimator (random forest)
3. Run the nCV to get the mean accuracy score

To do this, my code so far is:

X_train, X_test, Y_train, Y_test = train_test_split(X_res, Y_res, test_size=0.2)
inner_cv = KFold(n_splits=2, shuffle=True)
outer_cv = KFold(n_splits=2, shuffle=True)
rfc = RandomForestClassifier()
param_grid = {'bootstrap': [True, False],
              'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None],
              'max_features': ['auto', 'sqrt', 'log2', None],
              'min_samples_leaf': [1, 2, 4, 25],
              'min_samples_split': [2, 5, 10, 25],
              'criterion': ['gini', 'entropy'],
              'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000]}
rfclf = RandomizedSearchCV(rfc, param_grid, cv=inner_cv, n_iter=100, n_jobs=-1, scoring='accuracy', verbose=1)
nested_cv_results = cross_val_score(rfclf, X_train, Y_trin, cv=outer_cv, scoring = 'accuracy')

I now have 2 questions:

1. How do I find the overall best model?
2. How do I test this the best model against X_test and Y_test?

Answer 1

As I understand it, performing nested CV means that you have two layers of CV, and you use each layer for one and only one thing:

• The outer layer, for estimating the quality of the models trained on the inner layer.
• The inner layer for selecting the best model (including parameters, hyperparameters and so on).

One important distinction: in fact, you are not assessing the quality of the model, but of the procedure of model selection. What you are evaluating in the outer folds is that the procedure for model selection is consistent, and the quality of the estimators built on the outer loops

So, for each k-outer-fold, you will select one (and only one) inner model (the best one), trained on the training set for this k-outer-fold, and this model will be evaluated on the validation set for this k-outer-fold. After you vary the outer k, you will have k-outer estimates and you can average them to assess better the quality of the models.

Another important thing: once you find the best hyperparameters for a particular loop, how can you select the model, if you have k-inner-folds models? You can do two things: create a new instance of the model with this parameters and fit it on the whole inner fold, or create an ensemble of all k-inner-fold models. Both can be ok.

I have released a package that can help implementing nested cross validation in Python (for the moment, it only works for binary classifiers). If you want to check it out, it's here:

https://github.com/JaimeArboleda/nestedcvtraining

It's my first Python package, so any comments, suggestions or critics will be more than welcome!!

I post it as an answer because nested cross validation is performed inside the main function and you don't have to take care of how to implement it. Anyway, the code is visible in my github account, so that you can check how I implemented it if you are curious. Also, the readme explain it in detail (although, maybe, not very clearly!).

After the function is called, you get a model (that in fact will be a pipeline if there is a post-process to perform) and a complete report of what happened inside, so that you can assess the estimates of the different scoring functions. It comes with many options thay may be enough for a lot of common settings, I hope.

I don't test with a completely separated test set, because the outer procedure is meant to do this repeatedly and you can assess the quality of the model. However, if you have any doubt I'll be more than happy to try to answer it.