Often deep learning/machine learning models are compared just by measuring their performance on a validation dataset - the model that performs better "wins".

Within null hypothesis testing, two samples are "truly" different only if the discrepancy is large enough to be "unlikely" if the null hypothesis were true.

How can I determine if the improvement is real rather than spurious? Can I just use bootstrap to calculate the CI for the accuracy (or mAP, or L2 norm, etc.), or are there better (more robust/sensitive) methods?

Edit: I'm specifically talking about deep learning models. It often takes days to train a DL model, so k-fold cross-validation (or really anything that requires to re-train the model) isn't a practical option.

Related issues:

  • What are the best practices to control for the effect of multiple comparisons? [e.g. if I try to beat the performance of F on the dataset X, sooner or later I will succeed - how can I avoid doing that for the wrong reason?]

  • Are there practices that "help" not getting stuck during manual hyperparameter search? [e.g. after having found the optimal LR, number of layers/filters, etc. for a given optimization algorithm (e.g. SGD) if I replace it with a new one (e.g. Adam) I think might observe a drop in the performance even if the latter was a better choice, because of the co-dependence between the different hyperparameter]


This is a common problem in machine learning. In a very general scenario you have $M$ models that you want to compare on $D$ datasets using $k$-fold cross-validation. The performance measure doesn't really matter, as long it is a scalar value, accuracy, area under the ROC, F1-measure, etc. all would work. Naively, you would pool the results for each model across all datasets and all folds and compare the mean using e.g. ANOVA. However, this approach is flawed, because results from different folds for a single dataset are clearly not independent, because they have been derived from the same dataset.

There are two important papers that deal with this kind of situation. Demšar proposed a set of frequentist tests that account for correlations and provide a measure termed critical difference to determine whether any set of methods actually perform different from each other. These methods have been implemented in the scmamp R package. However, there is one big problem with frequentist tests: the p-value is a function of the dataset size. You could just run 100 instead of 10 datasets and surely the differences will be significant eventually. In addition, the null hypothesis that all classifiers perform exactly the same is highly unlikely to begin with.

Benavoli et al. developed a set of Bayesian tests that address this problem. These methods are preferred, because they allow to define your personal region of practical equivalence (ROPE) and avoids relying on unrealistic assumptions of the null hypothesis to determine significance. There is a notebook with code that illustrates this nicely.

| cite | improve this answer | |
  • $\begingroup$ That is a great answer but unfortunately it isn't applicable to deep networks - it takes to long to train them to use a k-fold approach. Do you have experience of using the holdout method (k=1) for model selection? $\endgroup$ – ajeje Aug 3 '18 at 2:43
  • $\begingroup$ On the plus side, the datasets are tipically large enough to make the partition process robust even with k=1. On the downside, k=1 changes the comparison from between two algorithms to a comparison between two specific instances of the algorithms (so we can't control for the effect of the random seed, or other instance-specific sources of randomness) - I'm not sure if this is important though. $\endgroup$ – ajeje Aug 3 '18 at 2:46
  • $\begingroup$ Even if you use just a single fold (k=1), you can use aforementioned methods. I'm note sure what would be best if you want to compare a single algorithm and single dataset, but each instance with a different random number seeds. $\endgroup$ – sebp Aug 3 '18 at 7:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.