Confusion on scikit-learn nested cross validation example

Question

There are a ton of threads on nested cross-validation. "An intuitive understanding of each fold of a nested cross validation for parameter/model tuning" gives a good explanation. scikit-learn has an example of what they refer to as nested cv, but it seems wrong. Am I misunderstanding the example?

The inner cv step in the example makes sense. Use the inner cv step to get the best estimator.

clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv)
clf.fit(X_iris, y_iris)
non_nested_scores[i] = clf.best_score_

The outer cv step does not. It's using the same data as the inner cv step, which means that at least some of the data that has been used for training in the inner cv loop will be used for scoring in the outer cv loop. In other words, the outer cv loop is computing an average performance of the inner cv model, with no consideration as to what data has been already used for training.

nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv)
nested_scores[i] = nested_score.mean()

I would expect the example to look like this:

# Loop for each trial
for i in range(NUM_TRIALS):
    outer_cv_scores = []
    inner_cv = KFold(n_splits=4, shuffle=True, random_state=i)

    for train, test in outer_cv.split(X_iris, y_iris):
        # Non_nested parameter search and scoring. Do not use any of the test data.
        clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv)
        clf.fit(X_iris[train], y_iris[train])

        # Score the trained model from the inner cv step on the test data for this outer fold.
        outer_cv_scores.append(scorer(clf, X=X_iris[test], y=y_iris[test]))

    # For the final score for this trial, average the scores across all outer cv folds.
    score_for_trial = outer_cv_scores.mean()

Have I found a bug in this example or am I misunderstanding something?

Answer 1

In short: The example is correct but not easy to follow. You understand things correctly around nested CV but you probably did not follow the example. I do not blame you; it took me a slow read to see it is right.

The example defines two K-Folds cross-validators. One called inner_cv and one called outer_cv. Notice that while both are simple 4-fold CV procedures they do not refer to the same data.

OK, let's go line by line now:

1. clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv) says: Fit the estimator svm via a parameter search using p_grid using the cross_validation procedure is based on the inner_cv cross-validator.
2. Inner CV object is then used explicitly through the line clf.fit(X_iris, y_iris); we take the X_iris dataset, all 150 instances of it, break it into 4 pieces as dictated by inner_cv in clf, do CV using those 4 pieces (folds) and fit our object.
3. clf.best_score_ then gives us the mean cross-validated score of the best estimator among the ones defined by hyperparameters p_grid.
4. cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv) says: Give us the scores of the estimator clf for each run of the cross validation defined by outer_cv. To do that, cross_val_score will go ahead and do the following:
   1. Take the X_iris dataset, all 150 instances of it, break it into 4 pieces (this is our outer CV).
   2. In the three pieces that are defined by the the outer_CV (~113 instances) as our training and validation set, train the clf estimator (the svm) by taking a piece of ~85 instances and validating the performance of that estimator using a piece of ~28 instances. After doing K (inner) repeats, return the best estimator among the ones defined by hyperparameters p_grid. (that is our inner CV)
   3. In the remaining one piece (~37 instances) that is our unseen test fold, it will go ahead and estimate the associated cost. To do that, it will take the best estimator as provided by the inner loop, use it to predict on the value on the test fold, compare their goodness of fit and finish up.
   4. This is repeated K (outer) times.

As you see the command cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv) does exactly what you described in your code implementing nested CV. Just that function hides all the details; the primary detail being that clf is not really an previously optimised classifier object but rather a procedure where an inner procedure does a CV. To that extent, if cross_val_score was presented with an object that had the best parameters from GridSearchCV but did not do any internal CV itself, cross_val_score would effectively give us the same results as GridSearchCV.