1
$\begingroup$

I am thinking of using cross validation to select the best algorithm (e.g. SVMs, Random Forest), and then doing parameter tuning on the selected algorithm to build a model.

Is it acceptable and how it is different from nested cross validation, where both parameter tuning and performance evaluation are done before building a model?

$\endgroup$
3
$\begingroup$

The algorithm would typically just be considered another hyperparameter.

E.g. in the widely used scenario of grid search, you train a variety of models and select an optimal one. So in addition to training a variety of SVM with, say, varying cost parameter, you also train random forests and then select from that larger variety of models.

This selection or optimization can be done with the help of an (inner) cross validation.

The obtained model then needs to undergo verification (aka validation), which can be done with another independent "outer" cross validation: that's the nested cross validation.


You could also use a 3-level cross validation setup, where the outermost again performs verification of the final obtained model. The 2nd and 3rd level cross validation procedures would then be part of the training process, e.g. level two comparing the best random forest vs. the best SVM (and selecting accordingly), the innermost (3rd level) being the algorithm-specific hyperparameter tuning.

However, compared with the approach above with just 2 levels, here your innermost optimization of algorithm-specific hyperparameters has fewer test cases available and is thus subject to more uncertainty. Unless you have huge numbers of cases available, this can cause your whole optimization to fail (see figure 8 in the Cawley paper @FelixvanDoorn linked).

$\endgroup$
2
$\begingroup$

Using cross-validation for model selection is perfectly acceptable in general.

However, when you also intend to optimize hyperparameters using nested cross-validation is a better idea.

Nested CV relies on splitting the data in three sets, train, validation and test. Hyperparameter tuning and model evaluation are done on different sets.

In non-nested CV model evaluation and parameter tuning is done on the same sets. This might lead to overly optimistic performance estimates (I like to think of this as overfitting on the test set, but this not per se the best way to describe this)

The scikit-learn documentation contains a nice comparison between cross-validation and nested cross-validation

For more in depth reading, I suggest the paper that is referenced in this documentation

$\endgroup$
  • $\begingroup$ thanks. but can I do non-nested CV, then only do parameter tuning on the selected model? Is it acceptable? $\endgroup$ – Stuart Peterson Sep 22 '18 at 11:24
  • $\begingroup$ Yes you could. Do bear in mind that in order to properly evaluate the effects of the tuning, I would use different sets for evaluation and tuning of hyperparameters. $\endgroup$ – Felix van Doorn Sep 22 '18 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.