12
$\begingroup$

I often see people talking about 5x2 cross-validation as a special case of nested cross validation.

I assume the first number (here: 5) refers to the number of folds in the inner loop and the second number (here: 2) refers to the number of folds in the outer loop? So, how is this different from a "traditional" model selection and evaluation approach? By "traditional", I mean

  • split the dataset into a separate training (e.g., 80%) and test set
  • use k-fold cross-validation (e.g., k=10) for hyperparameter tuning and model selection on the training set
  • evaluate generalization performance of the selected model using the test set

Isn't 5x2 exactly the same except that the test and training set have equal size if k=2?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ You are correct, in this case it is the same, except that it uses a 50/50 split in the outer loop instead of a 80/20 one. Generally, it gives a better estimate of the generalization performance and should be preferred, especially with relatively small sample sizes. From my experience, even for nested CV, performance estimation varies a lot. Often it is better to perform nested CV multiple times to get a good estimate of generalization performance. $\endgroup$
    – George
    Commented May 11, 2015 at 8:47
  • $\begingroup$ Thanks, makes sense! However, for small training sets, I'd probably increase the number of folds in the inner and outer loops; might decrease the variance but also increase the bias though $\endgroup$
    – user39663
    Commented May 11, 2015 at 12:49
  • $\begingroup$ In general, instead of doing a 5x2 nested CV, I usually perform a (k-1)xk, with k = 5 or 10. In case of few samples, instead of increasing the number of folds I would go for smaller values of k. $\endgroup$
    – George
    Commented May 11, 2015 at 13:37
  • 2
    $\begingroup$ I think you had it backwards rather than completely wrong, but the accepted answer may disagree with the source with which I'm about to refer. In Python Machine Learning by Raschka, he refers to a, "particular type of nested cross-validation is also known as 5x2 cross-validation." There's an included graphic in which he shows that the 2 refers to the inner loop for hyper parameter tuning and the 5 refers to the outer loop for unbiased model performance estimation. A colored copy of the graphic can be found under Scenario 3 here: sebastianraschka.com/faq/docs/evaluate-a-model.html $\endgroup$
    – Austin
    Commented Aug 30, 2017 at 0:38

2 Answers 2

Reset to default
16
$\begingroup$

5x2cv as far as I have seen in the literature, always refer to a 5 repetition of a 2-fold. There is no nesting at all. do a 2-fold (50/50 split between train and test), repeat it 4 more times. The 5x2cv was popularised by the paper Approximate statistical tests for comparing supervised classification learning algorithms by Dietterich as a way of obtaining not only a good estimate of the generalisation error but also a good estimate of the variance of that error (in order to perform statistical tests)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! Do you know what people commonly do if the inner loops select different models, e.g., if the "optimal" regularization parameter is lambda=100 during one model selection and lambda=1000 for the other one? In this case calculating the average model performance would be a little bit weird, right!? Would you discard the models as being "unstable"? $\endgroup$
    – user39663
    Commented May 11, 2015 at 12:57
  • 3
    $\begingroup$ The inner loop will very likely result in different selection of hyperparameters. You do not use nested cross-validation to select the hyperparameters, only to get a good estimate of the generalization error (with the best hyperparameters possible). Nested cv is used to decide between one or another algorithm. See stats.stackexchange.com/questions/136296/… or stats.stackexchange.com/questions/65128/… (among others) $\endgroup$
    – Jacques Wainer
    Commented May 11, 2015 at 14:54
  • $\begingroup$ Oh, I see, that makes complete sense then! I thought people were using it differently. I think we I can close the question then. $\endgroup$
    – user39663
    Commented May 11, 2015 at 17:32
3
$\begingroup$

2 repetitions in outer loop mean that you repeat your 5-fold CV 2 times on the whole train set. Each time subdivision into folds will be different.

This is mainly used for better estimations of model performance, like running statistical tests on whether one model performs statistically-significantly better than another.

Nested CV is not critically important if your data set is large and without outliers. If your data do have outliers, than cross validation performance may be drastically different depending on what fold/folds these outliers are in. Therefore you repeat CV several times.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Good point. In the traditional approach (test/train split and then k-fold CV on the training set) you only have 1 fold for evaluating the model whereas in 5x2 CV the average performance can be calculated from the 2 different folds. $\endgroup$
    – user39663
    Commented May 11, 2015 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.