Calculating Confidence Intervals for Cross Validated Binary Classifiers

Question

I'm experimenting with a number of models for a binary classification problem. To evaluate the performance of each model, I've used 10x repeated 10-fold cross validation to calculate the PR AUC (Area Under the Precision-Recall Curve), and a number of other metrics.

My professor would like me to report the confidence intervals for these metrics and the p-value of the conclusion that model X is better than model Y. This is standard practice in Statistical Machine Translation (SMT) experiments (his background), where they use bootstrapping to calculate these (my experiment is NLP-related, but not SMT). Since I have used repeated cross-validation to estimate each model's performance, I'm not sure how to calculate the confidence intervals or p-values.

This question is similar, where they suggest bootstrapping the resampled mean, but I'm not sure what this means. Is this the correct approach, and, if so, can anyone explain how to do that in more detail?

Answer 1

Summary: Whatever you do in order to calculate confidence intervals based on repeated CV, you need to take into account that there are several different sources of uncertainty.

---

Long version: Let me add my 2 ct with regard to repeated cross validation:

Repeated cross validation allows you to separate 2 sources of variance uncertainty in the test results.

1. variance due to model instability, i.e. variance in the predictions for the same case by different surrogate models (i.e. exchange a few training cases in the surrogate models' training sets) and
2. variance due to the finite (limited) number of independent test cases

Now typically cross validation assumes that the models are stable, i.e. variance 1. is negligible. This assumption you can easily check. If you find it non-negligible, you'd typically go back to training and try to stabilize your models before doing anything else.
Variance 2 depends heavily on the total number of independent cases = the total number of independent cases tested in each run of the cross validation (and is usually much worse if you insist on classification rather than staying with metric scores).

I'm pointing this out because repetitions of the cross validation can help estimating and reducing variance 1, but will not mitigate variance 2 - but under the standard assumptions for cross validation, variance 2 should be dominating.
I suspect that this the underlying cause for VanWinckelen's finding that "Repeated cross-validation should not be assumed to give much more precise estimates of a model’s predic- tive accuracy".

If, instead of characterizing the model you get using the data at hand, you try to find out whether one or the other algorithm would be better for similar applications, you have more unknown sources of uncertainty, see Bengio and Grandvalet: No unbiased estimator of the variance of k-fold cross-validation.

---

Disclaimer: I cannot say much on the area under those curves as for my applications the curves are often skewed and I rather need to take into account pairs of figures of merit such as sens and spec or PPV and NPV (or scoring rules that are analogous to them).

Answer 2

Since you are computing areas, note that the AUROC is just the concordance probability $c$ between predicted risks and observed binary outcomes. And so you are not needing to engage in classification. Instead your outcomes can be just predicted risks. There are methods for getting confidence intervals for $c$-indexes by computing a standard error for $c$. But $c$ (AUROC) is not sensitive enough for there to be good statistical power for comparing two models. For that use the gold standard likelihood ratio $\chi^2$ test (if the models are nested) or compute the difference in two proper probability accuracy scores. You might also look into AIC for informal comparisons of non-nested models.

Formal $\chi^2$ or $F$ tests are done using pre-specified models and do not involve any resampling such as cross-validation. When more complex situations are present and you use a resampling procedure to unbiasedly estimate an accuracy score, things are more complicated and we don't have all the theory worked out yet. An option is always there - to use the double bootstrap, which is computationally intensive.