5
$\begingroup$

I have been reading the questions related to nested cross validation and model selection and also gone through some tutorials. But I still do not understand how to solve the following problem: Suppose I have 2 classifiers: Logistic regression and Neural Network. I have some data (say 10000). I need to first find the best hyper-parameters for both of the classifiers and then also compare their performance. Here is what I think is reasonable:

  1. Create training set with 800 data points, keep 200 for testing
  2. Use k-fold cross validation with each of the classifiers to find the best hyper-parameters (such as regularizer or number of hidden nodes)
  3. Train both classifiers with total 800 data points and use the 200 data points for comparing the two classifiers.

I do not know if those steps are according to any standard procedure.

In several tutorials I found the process called nested CV, and here I get confused. If I use an outer loop for model comparison, and inner loop to select best parameter, then at each outer iteration, different hyper-parameter might be selected. But I want to find only one (the best) hyper parameter once and then compare the classifiers.

My questions are

  1. Are the previously mentioned steps follow any standard procedure?
  2. If not, how can I use Repeated/Nested CV in my case?
  3. I also want to do statistical analysis on the accuracy of the 3 classifiers (e.g. t test), how should I do that?

Thanks in advance.

$\endgroup$
6
  • 1
    $\begingroup$ What you describe (k-fold CV on the training data and then evaluation of test set) if fine. If anything I would suggested repeated $k$-fold CV to get a bit more stable estimates. $\endgroup$
    – usεr11852
    Dec 7 '16 at 22:57
  • $\begingroup$ thanks @usεr11852. could you please refer some tutorials on repeated k-fold CV? $\endgroup$
    – Rakib
    Dec 7 '16 at 23:04
  • $\begingroup$ Sorry no tutorials spring to mind. Molinaro's 2005 paper on : “Prediction Error Estimation: A Comparison of Resampling Methods.” is pretty straightforward I think though. You might want to also check: Kim's 2009: “Estimating Classification Error Rate: Repeated Cross-Validation, Repeated Hold-Out and Bootstrap.” This thread should be helpful too. $\endgroup$
    – usεr11852
    Dec 7 '16 at 23:20
  • 1
    $\begingroup$ @usεr11852 : a bit of an anti thesis but in addition to computation, repeated CV has other costs : On Estimating Model Accuracy with Repeated Cross-Validation (Vanwinckelen et. al. 2011). lirias.kuleuven.be/bitstream/123456789/346385/3/… $\endgroup$
    – discipulus
    Dec 9 '16 at 1:22
  • $\begingroup$ @discipulus: +1. Thank you for the paper. I was unaware of it! Through a quick read the paper what I take back is that rep-CV is no panacea. Not that it is wrong and should not be done. It showcases that people should not (and indeed never should) treat rep-CV as a Holy Grail. (Also I do find the methodology of the paper a bit peculiar: Treating samples of less that 10k as a populations and then saying that rep-CV does not get the "true-population-value" seems a bit of stretch. Similarly folds of $N$ = 200 are quite small; their 1k folds fare obviously much better and are more realistic.) $\endgroup$
    – usεr11852
    Dec 9 '16 at 21:03
2
$\begingroup$

Are the previously mentioned steps follow any standard procedure? Yes! You are using hold-out validation set for final classifier comparison and k-fold cross-validation for the parameter (model) selection.

If not, how can I use Repeated/Nested CV in my case? Since you are considering different models, one way to improve that would be :

For each method

  1. Use k-fold cross-validation for model selection
  2. After selecting the optimal parameters (model fitting), use k-fold cross-validation to get the generalisation error.

This gives you the variation in errors in different folds, so you can calculate, variance (or standard deviation) to report on the reliability/consistency of the model, or even generate some plot.

UPDATE

You don't need to split the data for step 1 and step 2. Use 10000 data points in k-fold cross-validation, i.e., if k = 10, then you will use 9000 for training and 1000 for validation for model selection. Once model is selected again use the same 10000 samples in the similar k-fold cross-validation but this time your parameters will be fixed.

You can choose to run k-fold cross-validation once and get k error measures for each of the subset; 2*k if you consider training set which you could also look into. So, with those k or 2*k values you can perform some statistical tests or draw some plots. It is also good to repeat the cross-validation process n times, giving you n *k error measures for statistical analysis.

$\endgroup$
4
  • $\begingroup$ Thanks. So your suggestion is (if I understood correctly), lets say I have 10000 data, then split it (what should be the ratio?) in two sub sets. use the first subset for cross-validation to select hyper parameters for each classifier. Then use the second subset for testing the final classifiers, but instead of using all at once, divide it further, test on each sub-subset and get average error and variance (how?) ? I also want to do statistical analysis on the accuracy of the 3 classifiers (e.g. t test), how should I do that? $\endgroup$
    – Rakib
    Dec 8 '16 at 0:20
  • $\begingroup$ See my updated answer. $\endgroup$
    – discipulus
    Dec 8 '16 at 1:16
  • 1
    $\begingroup$ If you do model fitting (hyper-parameter tuning) on the same data as the data used to estimate generalization error (for comparing two classifiers) then you will overfit to the data and your generalization error will be optimistic. Doing what Rakib suggested - splitting the data into two sub sets - is one way to avoid this. $\endgroup$
    – MD004
    Jan 29 '19 at 19:40
  • $\begingroup$ @MD004: Your point is valid because we have already used that data in hyper-parameter selection. If you have larger dataset, we can run cross-validation on independent sets. Nevertheless, we are using the process as independent processes. First identify hyper-parameters. Second, again retrain in cross-validation. For some cases when data is scarce, we are not leaking training and test data and also the assumption of any ML algorithm is that future data are similar to the current one but not necessarily the same. So, assumption could still hold. $\endgroup$
    – discipulus
    Feb 2 '19 at 4:33
1
$\begingroup$

I'd like to propose an argument, that @discipulus answer is not entirely correct.

What is the standard procedure?

Normally, the setup looks like this:

  1. Split the dataset (for example, training 60%, cross-validation 20%, test 20%).
  2. [Cross-validation set] Find the best model (comparing different models and/or different hyperparameters for each). Model selection ends with this step.
  3. [Test set] Get an estimate of how the model might perform in "the real world".

Caveats

  • If you don't need to compare models, and don't need to optimize hyperparameters for those models, you can skip step 2 and not allocate the cross-validation subset (20% in our case).
  • If you don't need an estimate of the actual performance in the real world, you can skip step 3 and not allocate the test subset (20% in our case).
  • Do not choose the model based on the test set performance. Let's imagine that on cross-validation (step 2), model A (with some specific hyperparameters) gets 90% accuracy, and model B (with some specific hyperparameters) gets 80%. Now, let's say you got curious and ran both models on the test set (step 3), and the results are model A gets 80%, while model B gets 90% (opposite than before). What to do? Use only the cross-validation results to select the model, i.e. the correct answer here is to use the model A (Related answer). Why I can't choose based on test set? Because you'd be essentially selecting some model out of many models, and you might get lucky to find the one which just so happens to perform well on the test set, therefore you will not be able to trust your test set accuracy anymore. More detailed explanation is available here.

Applied to your example

You use step 2 and step 3 for exactly the same reason - selecting the best model and it's hyperparameters combination. You could have your setup like this:

  1. Create training set with 800 data points, keep 200 for cross-validation (no test set, since you didn't mentioned you want to evaluate estimate of the "real world" performance).
  2. Use the cross-validation dataset with each of the classifiers to find the best hyper-parameters (such as regularizer or number of hidden nodes) Let's say your results are:
  • Model1: LogisticRegression, regularizer=0.1, accuracy 80%
  • Model2: LogisticRegression, regularizer=0.01, accuracy 80%
  • Model3: LogisticRegression, regularizer=0.001, accuracy 81%
  • Model4: NeuralNetwork, hidden_nodes=5, accuracy 71%
  • Model5: NeuralNetwork, hidden_nodes=10, accuracy 82%
  • Model6: NeuralNetwork, hidden_nodes=25, accuracy 76%

That's it, NeuralNetwork is the better model than LogisticRegression

What if I want to evaluate performance? You can't use the 82% as your accuracy estimate "in the real world", because it now has optimistic bias, due to you selecting for it. If you'd want to have an estimate of how your model performs, you need to add the 3rd step as described in the "standard procedure" section. In your setup it would look like this:

  1. Create training set with 600 data points, keep 200 for cross-validation, also keep 200 for test.
  2. [same actions, same results as previously].
  3. Train the NeuralNetwork with 10 hidden nodes on 800 data points (training set + cross-validation set) and test on the 200 data points (test set)

Repeatability: How to use nested k-fold cross validation

Imagine, you reshuffle your 1000 data points, then do the step 2, and you get entirely different accuracies, and now the best model is LogisticRegression with regularizer=0.01. This is a problem since just by shuffling the dataset we got a different outcome.

One way of how to get a stable accuracy estimate would be to use k-fold cross validation for the step 2 (exactly as you described in your original post). But we could do k-fold cross validation for the step 3 as well, to get a better accuracy estimate. It would be called nested k-fold cross validation and would go like this:

Use k-fold cross validation (for example, if k=5, then the 1000 data points are split to `trainval` dataset with 800 data points, and `test` dataset with 200 data points).

FOR EACH of the 5 800+200 (trainval+test) datapoints splits {
    Take the `trainval` 800 datapoints and use k-fold cross validation (for example, if k=8, then the 800 datapoints are split to `train` dataset with 700 data points and `val` dataset with 100 data points

    FOR EACH of the 4 700+100 (train+val) splits {
        Train a model with some specific hyperparameters with 700 data points, then calculate accuracy with the 100 `val` set.
    }  

    Calculate accuracy of the best model+hyperparameter pair for the 200 datapoints.
}

You should have trained 3 (model+hyperparameter pairs) * 5 (outer cross-validation) * 4 (inner c-v) = 60 models.

More resources on Nested k-fold crossvalidation

  • There's an excellent blog post by Weina Jin, which includes more detailed description and implementation pseudo-code
  • Nested k-fold cross validation can be visualized like this (image source): Nested k-fold cross validation visualization
  • Pseudo-code is also available here, and here.
  • Here and here are quick summaries of the nested k-fold cross validation. Here is a longer one. Here is a bit more information on when nested k-fold validation is useful.

Regarding the t-test statistical analysis

This question alone could warrant a separate post on the stack exchange, but this Article explains why it might not be the best idea and among the suggestions is to use McNemar’s test or 5×2 Cross-Validation instead.

We could then select and use the paired Student’s t-test to check if the difference in the mean accuracy between the two models is statistically significant, e.g. reject the null hypothesis that assumes that the two samples have the same distribution. [...]

The problem is, a key assumption of the paired Student’s t-test has been violated.

Namely, the observations in each sample are not independent. As part of the k-fold cross-validation procedure, a given observation will be used in the training dataset (k-1) times. This means that the estimated skill scores are dependent, not independent, and in turn that the calculation of the t-statistic in the test will be misleadingly wrong along with any interpretations of the statistic and p-value.

Regarding reporting deviations and confidence intervals

It might not be the best choice as well.

There appears to be some confusion among researchers, however, about best practices for cross-validation, and about the interpretation of cross-validation results. In particular, [...] standard deviations, confidence intervals, or an indication of ”significance”.

In this paper, we argue that, under many practical circumstances, when the goal of the experiments is to see how well the model returned by a learner will perform in practice in a particular domain, repeated cross-validation is not useful, and the reporting of confidence intervals or significance is misleading.

Source: On Estimating Model Accuracy with Repeated Cross-Validation. Gitte Vanwinckelen, Hendrik Blockeel. Department of Computer Science, KU Leuven; Heverlee, Belgium.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.