My deep learning model improved - is the difference significant?

Question

Often deep learning/machine learning models are compared just by measuring their performance on a validation dataset - the model that performs better "wins".

Within null hypothesis testing, two samples are "truly" different only if the discrepancy is large enough to be "unlikely" if the null hypothesis were true.

How can I determine if the improvement is real rather than spurious? Can I just use bootstrap to calculate the CI for the accuracy (or mAP, or L2 norm, etc.), or are there better (more robust/sensitive) methods?

Edit: I'm specifically talking about deep learning models. It often takes days to train a DL model, so k-fold cross-validation (or really anything that requires to re-train the model) isn't a practical option.

---

Related issues:

• What are the best practices to control for the effect of multiple comparisons? [e.g. if I try to beat the performance of F on the dataset X, sooner or later I will succeed - how can I avoid doing that for the wrong reason?]

• Are there practices that "help" not getting stuck during manual hyperparameter search? [e.g. after having found the optimal LR, number of layers/filters, etc. for a given optimization algorithm (e.g. SGD) if I replace it with a new one (e.g. Adam) I think might observe a drop in the performance even if the latter was a better choice, because of the co-dependence between the different hyperparameter]

Answer 1

This is a common problem in machine learning. In a very general scenario you have $M$ models that you want to compare on $D$ datasets using $k$-fold cross-validation. The performance measure doesn't really matter, as long it is a scalar value, accuracy, area under the ROC, F1-measure, etc. all would work. Naively, you would pool the results for each model across all datasets and all folds and compare the mean using e.g. ANOVA. However, this approach is flawed, because results from different folds for a single dataset are clearly not independent, because they have been derived from the same dataset.

There are two important papers that deal with this kind of situation. Demšar proposed a set of frequentist tests that account for correlations and provide a measure termed critical difference to determine whether any set of methods actually perform different from each other. These methods have been implemented in the scmamp R package. However, there is one big problem with frequentist tests: the p-value is a function of the dataset size. You could just run 100 instead of 10 datasets and surely the differences will be significant eventually. In addition, the null hypothesis that all classifiers perform exactly the same is highly unlikely to begin with.

Benavoli et al. developed a set of Bayesian tests that address this problem. These methods are preferred, because they allow to define your personal region of practical equivalence (ROPE) and avoids relying on unrealistic assumptions of the null hypothesis to determine significance. There is a notebook with code that illustrates this nicely.