3
$\begingroup$

In the image that I am attaching says that a held-out set is needed for feature selection. I know that for feature selection a nested cross-validation is needed. So if I apply "pipeline" I do not need to create an outer cross-validation and neither an inner one. Am I right?

http://scikit-learn.org/stable/modules/cross_validation.html enter image description here

$\endgroup$

1 Answer 1

5
$\begingroup$

The cross_val_score() function applies Cross Validation to obtain an unbiased measure of the accuracy of a classifier (or pipeline in sklearn).

In the example you provided, the second block of code performs a 3-fold cross validation. The whole dataset $X$ is randomly divided into 3 non-overlaping subsets $X_1, X_2, X_3$. Then, the following experiments are automatically performed:

  1. Train a StandarScaler on $X_1,X_2$ and use it to transform $X_1, X_2, X_3$. Train a SVM on the scaled versions of $X_1,X_2$, test on the scaled version of $X_3$.

  2. Train a StandarScaler on $X_1,X_3$ and use it to transform $X_1, X_2, X_3$. Train a SVM on the scaled versions of $X_1,X_3$, test on the scaled version of $X_2$.

  3. Train a StandarScaler on $X_2,X_3$ and use it to transform $X_1, X_2, X_3$. Train a SVM on the scaled versions of $X_2,X_3$, test on the scaled version of $X_1$.

The accuracies obtained on these three experiments can be averaged to obtain an unbiased estimate of the generalization accuracy of your pipeline: StandardScaller() + SVC(C=1). However, if you started playing with the hyperparameters of your classifier (e.g. trying different C values) the best accuracy obtained would not be an unbiased estimate.

Nested CV is necesary if you want to perform hyperparametrer/model selection (inner loop) and obtain an unbiased estimate of the accuracy of your method (outer loop).

These topics have been already discused in more detail here and here. In short: "the outer loop is to assess the performance of the model, and the inner loop is to select the best model".

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.