There are a ton of threads on nested cross-validation. "An intuitive understanding of each fold of a nested cross validation for parameter/model tuning" gives a good explanation. scikit-learn has an example of what they refer to as nested cv, but it seems wrong. Am I misunderstanding the example?
The inner cv step in the example makes sense. Use the inner cv step to get the best estimator.
clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv) clf.fit(X_iris, y_iris) non_nested_scores[i] = clf.best_score_
The outer cv step does not. It's using the same data as the inner cv step, which means that at least some of the data that has been used for training in the inner cv loop will be used for scoring in the outer cv loop. In other words, the outer cv loop is computing an average performance of the inner cv model, with no consideration as to what data has been already used for training.
nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv) nested_scores[i] = nested_score.mean()
I would expect the example to look like this:
# Loop for each trial for i in range(NUM_TRIALS): outer_cv_scores =  inner_cv = KFold(n_splits=4, shuffle=True, random_state=i) for train, test in outer_cv.split(X_iris, y_iris): # Non_nested parameter search and scoring. Do not use any of the test data. clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv) clf.fit(X_iris[train], y_iris[train]) # Score the trained model from the inner cv step on the test data for this outer fold. outer_cv_scores.append(scorer(clf, X=X_iris[test], y=y_iris[test])) # For the final score for this trial, average the scores across all outer cv folds. score_for_trial = outer_cv_scores.mean()
Have I found a bug in this example or am I misunderstanding something?