# How to use cross validation for model comparison

I have been reading the questions related to nested cross validation and model selection and also gone through some tutorials. But I still do not understand how to solve the following problem: Suppose I have 2 classifiers: Logistic regression and Neural Network. I have some data (say 10000). I need to first find the best hyper-parameters for both of the classifiers and then also compare their performance. Here is what I think is reasonable:

1. Create training set with 800 data points, keep 200 for testing
2. Use k-fold cross validation with each of the classifiers to find the best hyper-parameters (such as regularizer or number of hidden nodes)
3. Train both classifiers with total 800 data points and use the 200 data points for comparing the two classifiers.

I do not know if those steps are according to any standard procedure.

In several tutorials I found the process called nested CV, and here I get confused. If I use an outer loop for model comparison, and inner loop to select best parameter, then at each outer iteration, different hyper-parameter might be selected. But I want to find only one (the best) hyper parameter once and then compare the classifiers.

My questions are

1. Are the previously mentioned steps follow any standard procedure?
2. If not, how can I use Repeated/Nested CV in my case?
3. I also want to do statistical analysis on the accuracy of the 3 classifiers (e.g. t test), how should I do that?

• What you describe (k-fold CV on the training data and then evaluation of test set) if fine. If anything I would suggested repeated $k$-fold CV to get a bit more stable estimates. Dec 7, 2016 at 22:57
• thanks @usεr11852. could you please refer some tutorials on repeated k-fold CV? Dec 7, 2016 at 23:04
• Sorry no tutorials spring to mind. Molinaro's 2005 paper on : “Prediction Error Estimation: A Comparison of Resampling Methods.” is pretty straightforward I think though. You might want to also check: Kim's 2009: “Estimating Classification Error Rate: Repeated Cross-Validation, Repeated Hold-Out and Bootstrap.” This thread should be helpful too. Dec 7, 2016 at 23:20
• @usεr11852 : a bit of an anti thesis but in addition to computation, repeated CV has other costs : On Estimating Model Accuracy with Repeated Cross-Validation (Vanwinckelen et. al. 2011). lirias.kuleuven.be/bitstream/123456789/346385/3/… Dec 9, 2016 at 1:22
• @discipulus: +1. Thank you for the paper. I was unaware of it! Through a quick read the paper what I take back is that rep-CV is no panacea. Not that it is wrong and should not be done. It showcases that people should not (and indeed never should) treat rep-CV as a Holy Grail. (Also I do find the methodology of the paper a bit peculiar: Treating samples of less that 10k as a populations and then saying that rep-CV does not get the "true-population-value" seems a bit of stretch. Similarly folds of $N$ = 200 are quite small; their 1k folds fare obviously much better and are more realistic.) Dec 9, 2016 at 21:03

Are the previously mentioned steps follow any standard procedure? Yes! You are using hold-out validation set for final classifier comparison and k-fold cross-validation for the parameter (model) selection.

If not, how can I use Repeated/Nested CV in my case? Since you are considering different models, one way to improve that would be :

For each method

1. Use k-fold cross-validation for model selection
2. After selecting the optimal parameters (model fitting), use k-fold cross-validation to get the generalisation error.

This gives you the variation in errors in different folds, so you can calculate, variance (or standard deviation) to report on the reliability/consistency of the model, or even generate some plot.

UPDATE

You don't need to split the data for step 1 and step 2. Use 10000 data points in k-fold cross-validation, i.e., if k = 10, then you will use 9000 for training and 1000 for validation for model selection. Once model is selected again use the same 10000 samples in the similar k-fold cross-validation but this time your parameters will be fixed.

You can choose to run k-fold cross-validation once and get k error measures for each of the subset; 2*k if you consider training set which you could also look into. So, with those k or 2*k values you can perform some statistical tests or draw some plots. It is also good to repeat the cross-validation process n times, giving you n *k error measures for statistical analysis.

• Thanks. So your suggestion is (if I understood correctly), lets say I have 10000 data, then split it (what should be the ratio?) in two sub sets. use the first subset for cross-validation to select hyper parameters for each classifier. Then use the second subset for testing the final classifiers, but instead of using all at once, divide it further, test on each sub-subset and get average error and variance (how?) ? I also want to do statistical analysis on the accuracy of the 3 classifiers (e.g. t test), how should I do that? Dec 8, 2016 at 0:20
• See my updated answer. Dec 8, 2016 at 1:16
• If you do model fitting (hyper-parameter tuning) on the same data as the data used to estimate generalization error (for comparing two classifiers) then you will overfit to the data and your generalization error will be optimistic. Doing what Rakib suggested - splitting the data into two sub sets - is one way to avoid this. Jan 29, 2019 at 19:40
• @MD004: Your point is valid because we have already used that data in hyper-parameter selection. If you have larger dataset, we can run cross-validation on independent sets. Nevertheless, we are using the process as independent processes. First identify hyper-parameters. Second, again retrain in cross-validation. For some cases when data is scarce, we are not leaking training and test data and also the assumption of any ML algorithm is that future data are similar to the current one but not necessarily the same. So, assumption could still hold. Feb 2, 2019 at 4:33

I'd like to propose an argument, that @discipulus answer is not entirely correct.

# What is the standard procedure?

Normally, the setup looks like this:

1. Split the dataset (for example, training 60%, cross-validation 20%, test 20%).
2. [Cross-validation set] Find the best model (comparing different models and/or different hyperparameters for each). Model selection ends with this step.
3. [Test set] Get an estimate of how the model might perform in "the real world".

### Caveats

• If you don't need to compare models, and don't need to optimize hyperparameters for those models, you can skip step 2 and not allocate the cross-validation subset (20% in our case).
• If you don't need an estimate of the actual performance in the real world, you can skip step 3 and not allocate the test subset (20% in our case).
• Do not choose the model based on the test set performance. Let's imagine that on cross-validation (step 2), model A (with some specific hyperparameters) gets 90% accuracy, and model B (with some specific hyperparameters) gets 80%. Now, let's say you got curious and ran both models on the test set (step 3), and the results are model A gets 80%, while model B gets 90% (opposite than before). What to do? Use only the cross-validation results to select the model, i.e. the correct answer here is to use the model A (Related answer). Why I can't choose based on test set? Because you'd be essentially selecting some model out of many models, and you might get lucky to find the one which just so happens to perform well on the test set, therefore you will not be able to trust your test set accuracy anymore. More detailed explanation is available here.

You use step 2 and step 3 for exactly the same reason - selecting the best model and it's hyperparameters combination. You could have your setup like this:

1. Create training set with 800 data points, keep 200 for cross-validation (no test set, since you didn't mentioned you want to evaluate estimate of the "real world" performance).
2. Use the cross-validation dataset with each of the classifiers to find the best hyper-parameters (such as regularizer or number of hidden nodes) Let's say your results are:
• Model1: LogisticRegression, regularizer=0.1, accuracy 80%
• Model2: LogisticRegression, regularizer=0.01, accuracy 80%
• Model3: LogisticRegression, regularizer=0.001, accuracy 81%
• Model4: NeuralNetwork, hidden_nodes=5, accuracy 71%
• Model5: NeuralNetwork, hidden_nodes=10, accuracy 82%
• Model6: NeuralNetwork, hidden_nodes=25, accuracy 76%

That's it, NeuralNetwork is the better model than LogisticRegression

What if I want to evaluate performance? You can't use the 82% as your accuracy estimate "in the real world", because it now has optimistic bias, due to you selecting for it. If you'd want to have an estimate of how your model performs, you need to add the 3rd step as described in the "standard procedure" section. In your setup it would look like this:

1. Create training set with 600 data points, keep 200 for cross-validation, also keep 200 for test.
2. [same actions, same results as previously].
3. Train the NeuralNetwork with 10 hidden nodes on 800 data points (training set + cross-validation set) and test on the 200 data points (test set)

# Repeatability: How to use nested k-fold cross validation

Imagine, you reshuffle your 1000 data points, then do the step 2, and you get entirely different accuracies, and now the best model is LogisticRegression with regularizer=0.01. This is a problem since just by shuffling the dataset we got a different outcome.

One way of how to get a stable accuracy estimate would be to use k-fold cross validation for the step 2 (exactly as you described in your original post). But we could do k-fold cross validation for the step 3 as well, to get a better accuracy estimate. It would be called nested k-fold cross validation and would go like this:

Use k-fold cross validation (for example, if k=5, then the 1000 data points are split to trainval dataset with 800 data points, and test dataset with 200 data points).

FOR EACH of the 5 800+200 (trainval+test) datapoints splits {
Take the trainval 800 datapoints and use k-fold cross validation (for example, if k=8, then the 800 datapoints are split to train dataset with 700 data points and val dataset with 100 data points

FOR EACH of the 4 700+100 (train+val) splits {
Train a model with some specific hyperparameters with 700 data points, then calculate accuracy with the 100 val set.
}

Calculate accuracy of the best model+hyperparameter pair for the 200 datapoints.
}


You should have trained 3 (model+hyperparameter pairs) * 5 (outer cross-validation) * 4 (inner c-v) = 60 models.

# More resources on Nested k-fold crossvalidation

• There's an excellent blog post by Weina Jin, which includes more detailed description and implementation pseudo-code
• Nested k-fold cross validation can be visualized like this (image source):
• Pseudo-code is also available here, and here.
• Here and here are quick summaries of the nested k-fold cross validation. Here is a longer one. Here is a bit more information on when nested k-fold validation is useful.

# Regarding the t-test statistical analysis

This question alone could warrant a separate post on the stack exchange, but this Article explains why it might not be the best idea and among the suggestions is to use McNemar’s test or 5×2 Cross-Validation instead.

We could then select and use the paired Student’s t-test to check if the difference in the mean accuracy between the two models is statistically significant, e.g. reject the null hypothesis that assumes that the two samples have the same distribution. [...]

The problem is, a key assumption of the paired Student’s t-test has been violated.

Namely, the observations in each sample are not independent. As part of the k-fold cross-validation procedure, a given observation will be used in the training dataset (k-1) times. This means that the estimated skill scores are dependent, not independent, and in turn that the calculation of the t-statistic in the test will be misleadingly wrong along with any interpretations of the statistic and p-value.

# Regarding reporting deviations and confidence intervals

It might not be the best choice as well.

There appears to be some confusion among researchers, however, about best practices for cross-validation, and about the interpretation of cross-validation results. In particular, [...] standard deviations, confidence intervals, or an indication of ”significance”.

In this paper, we argue that, under many practical circumstances, when the goal of the experiments is to see how well the model returned by a learner will perform in practice in a particular domain, repeated cross-validation is not useful, and the reporting of confidence intervals or significance is misleading.