3
$\begingroup$

I'm trying to work my head around the example of Nested vs. Non-Nested CV in Sklearn. I checked multiple answers but I am still confused on the example. To my knowledge, a nested CV aims to use a different subset of data to select the best parameters of a classifier (e.g. C in SVM) and validate its performance. Therefore, from a dataset X, the outer 10-folds CV (for simplicity n=10) creates 10 training sets and 10 test sets:

(Tr0, Te0),..., (Tr0, Te9)

Then, the inner 10-CV splits EACH outer training set into 10 training and 10 test sets:

From Tr0: (Tr0_0,Te_0_0), ... , (Tr0_9,Te0_9)
From Tr9: (Tr9_0,Te_9_0), ... , (Tr9_9,Te9_9)

Now, using the inner CV, we can find the best values of C for every single outer Training set. This is done by testing all the possible values of C with the inner CV. The value providing the highest performance (e.g. accuracy) is chosen for that specific outer Training set. Finally, having discovered the best C values for every outer Training set, we can calculate an unbiased accuracy using the outer Test sets. With this procedure, the samples used to identify the best parameter (i.e. C) are not used to compute the performance of the classifier, hence we have a totally unbiased validation.

The example provided in the Sklearn page is:

inner_cv = KFold(n_splits=4, shuffle=True, random_state=i)
outer_cv = KFold(n_splits=4, shuffle=True, random_state=i)

# Non_nested parameter search and scoring
clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv)
clf.fit(X_iris, y_iris)
non_nested_scores[i] = clf.best_score_

# Nested CV with parameter optimization
nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv)
nested_scores[i] = nested_score.mean()

From what I understand, the code simply calculates the scores using two different cross-validations (i.e. different splits into training and test set). Both of them used the entire dataset. The GridCV identifies the best parameters using one (of the two CVs), then cross_val_score calculates, with the second CV, the performance when using the best parameters.

Am I interpreting a Nested CV in the wrong way? What am I missing from the example?

$\endgroup$
1
$\begingroup$

No the code does not calculate the scores using different cv. In fact both use the same cv but the nested cv includes hyperparameter tuning inside a second cv (inner cv) which allows to get a less biased estimate of the performance. The inner cv is performed by GridSearchCV, this is basically a classifier (clf) which uses always the best clf of the given params (the best params are evaluated through an internally cv step -> the inner cv).

If this clf is used alone (non-nested) an overestimation of the performance can happen as the optimization process is learning on the test data (The sklearn example output demonstrates this). Therefore, the performance of the optimization process is also evaluated through a second cv (outer cv) which is performed with cross_val_score.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.