How to use cross validation for model comparison

Question

I have been reading the questions related to nested cross validation and model selection and also gone through some tutorials. But I still do not understand how to solve the following problem: Suppose I have 2 classifiers: Logistic regression and Neural Network. I have some data (say 10000). I need to first find the best hyper-parameters for both of the classifiers and then also compare their performance. Here is what I think is reasonable:

1. Create training set with 800 data points, keep 200 for testing
2. Use k-fold cross validation with each of the classifiers to find the best hyper-parameters (such as regularizer or number of hidden nodes)
3. Train both classifiers with total 800 data points and use the 200 data points for comparing the two classifiers.

I do not know if those steps are according to any standard procedure.

In several tutorials I found the process called nested CV, and here I get confused. If I use an outer loop for model comparison, and inner loop to select best parameter, then at each outer iteration, different hyper-parameter might be selected. But I want to find only one (the best) hyper parameter once and then compare the classifiers.

My questions are

1. Are the previously mentioned steps follow any standard procedure?
2. If not, how can I use Repeated/Nested CV in my case?
3. I also want to do statistical analysis on the accuracy of the 3 classifiers (e.g. t test), how should I do that?

Thanks in advance.

Answer 1

Are the previously mentioned steps follow any standard procedure? Yes! You are using hold-out validation set for final classifier comparison and k-fold cross-validation for the parameter (model) selection.

If not, how can I use Repeated/Nested CV in my case? Since you are considering different models, one way to improve that would be :

For each method

1. Use k-fold cross-validation for model selection
2. After selecting the optimal parameters (model fitting), use k-fold cross-validation to get the generalisation error.

This gives you the variation in errors in different folds, so you can calculate, variance (or standard deviation) to report on the reliability/consistency of the model, or even generate some plot.

UPDATE

You don't need to split the data for step 1 and step 2. Use 10000 data points in k-fold cross-validation, i.e., if k = 10, then you will use 9000 for training and 1000 for validation for model selection. Once model is selected again use the same 10000 samples in the similar k-fold cross-validation but this time your parameters will be fixed.

You can choose to run k-fold cross-validation once and get k error measures for each of the subset; 2*k if you consider training set which you could also look into. So, with those k or 2*k values you can perform some statistical tests or draw some plots. It is also good to repeat the cross-validation process n times, giving you n *k error measures for statistical analysis.

Answer 2

I'd like to propose an argument, that @discipulus answer is not entirely correct.

What is the standard procedure?

Normally, the setup looks like this:

1. Split the dataset (for example, training 60%, cross-validation 20%, test 20%).
2. [Cross-validation set] Find the best model (comparing different models and/or different hyperparameters for each). Model selection ends with this step.
3. [Test set] Get an estimate of how the model might perform in "the real world".

Caveats

• If you don't need to compare models, and don't need to optimize hyperparameters for those models, you can skip step 2 and not allocate the cross-validation subset (20% in our case).
• If you don't need an estimate of the actual performance in the real world, you can skip step 3 and not allocate the test subset (20% in our case).
• Do not choose the model based on the test set performance. Let's imagine that on cross-validation (step 2), model A (with some specific hyperparameters) gets 90% accuracy, and model B (with some specific hyperparameters) gets 80%. Now, let's say you got curious and ran both models on the test set (step 3), and the results are model A gets 80%, while model B gets 90% (opposite than before). What to do? Use only the cross-validation results to select the model, i.e. the correct answer here is to use the model A (Related answer). Why I can't choose based on test set? Because you'd be essentially selecting some model out of many models, and you might get lucky to find the one which just so happens to perform well on the test set, therefore you will not be able to trust your test set accuracy anymore. More detailed explanation is available here.

Applied to your example

You use step 2 and step 3 for exactly the same reason - selecting the best model and it's hyperparameters combination. You could have your setup like this:

1. Create training set with 800 data points, keep 200 for cross-validation (no test set, since you didn't mention you want to get an estimate of the "real world" performance).
2. Use the cross-validation dataset with each of the classifiers to find the best hyper-parameters (such as regularizer or number of hidden nodes) Let's say your results are:

• Model1: LogisticRegression, regularizer=0.1, accuracy 80%
• Model2: LogisticRegression, regularizer=0.01, accuracy 80%
• Model3: LogisticRegression, regularizer=0.001, accuracy 81%
• Model4: NeuralNetwork, hidden_nodes=5, accuracy 71%
• Model5: NeuralNetwork, hidden_nodes=10, accuracy 82%
• Model6: NeuralNetwork, hidden_nodes=25, accuracy 76%

That's it, NeuralNetwork is the better model than LogisticRegression

What if I want to evaluate performance? You can't use the 82% as your accuracy estimate "in the real world", because it now has optimistic bias, due to you selecting for it. If you'd want to have an estimate of how your model performs, you need to add the 3rd step as described in the "standard procedure" section. In your setup it would look like this:

1. Create training set with 600 data points, keep 200 for cross-validation, also keep 200 for test.
2. [same actions, same results as previously].
3. Train the NeuralNetwork with 10 hidden nodes on 800 data points (training set + cross-validation set) and test on the 200 data points (test set)

Repeatability: How to use nested k-fold cross validation

Imagine, you reshuffle your 1000 data points, then do the step 2, and you get entirely different accuracies, and now the best model is LogisticRegression with regularizer=0.01. This is a problem since just by shuffling the dataset we got a different outcome.

One way of how to get a stable accuracy estimate would be to use k-fold cross validation for the step 2 (exactly as you described in your original post). But we could do k-fold cross validation for the step 3 as well, to get a better accuracy estimate. It would be called nested k-fold cross validation and would go like this:

Use k-fold cross validation (for example, if k=5, then the 1000 data points are split to `trainval` dataset with 800 data points, and `test` dataset with 200 data points).

FOR EACH of the 5 800+200 (trainval+test) datapoints splits {
    Take the `trainval` 800 datapoints and use k-fold cross validation (for example, if k=8, then the 800 datapoints are split to `train` dataset with 700 data points and `val` dataset with 100 data points

    FOR EACH of the 4 700+100 (train+val) splits {
        Train a model with some specific hyperparameters with 700 data points, then calculate accuracy with the 100 `val` set.
    }  

    Calculate accuracy of the best model+hyperparameter pair for the 200 datapoints.
}

You should have trained 3 (model+hyperparameter pairs) * 5 (outer cross-validation) * 4 (inner c-v) = 60 models.

More resources on Nested k-fold crossvalidation

• There's an excellent blog post by Weina Jin, which includes more detailed description and implementation pseudo-code
• Nested k-fold cross validation can be visualized like this (image source):
• Pseudo-code is also available here, and here.
• Here and here are quick summaries of the nested k-fold cross validation. Here is a longer one. Here is a bit more information on when nested k-fold validation is useful.

Regarding the t-test statistical analysis

This question alone could warrant a separate post on the stack exchange, but this Article explains why it might not be the best idea and among the suggestions is to use McNemar’s test or 5×2 Cross-Validation instead.

We could then select and use the paired Student’s t-test to check if the difference in the mean accuracy between the two models is statistically significant, e.g. reject the null hypothesis that assumes that the two samples have the same distribution. [...]

The problem is, a key assumption of the paired Student’s t-test has been violated.

Namely, the observations in each sample are not independent. As part of the k-fold cross-validation procedure, a given observation will be used in the training dataset (k-1) times. This means that the estimated skill scores are dependent, not independent, and in turn that the calculation of the t-statistic in the test will be misleadingly wrong along with any interpretations of the statistic and p-value.

Regarding reporting deviations and confidence intervals

It might not be the best choice as well.

There appears to be some confusion among researchers, however, about best practices for cross-validation, and about the interpretation of cross-validation results. In particular, [...] standard deviations, confidence intervals, or an indication of ”significance”.

In this paper, we argue that, under many practical circumstances, when the goal of the experiments is to see how well the model returned by a learner will perform in practice in a particular domain, repeated cross-validation is not useful, and the reporting of confidence intervals or significance is misleading.

Source: On Estimating Model Accuracy with Repeated Cross-Validation. Gitte Vanwinckelen, Hendrik Blockeel. Department of Computer Science, KU Leuven; Heverlee, Belgium.