Calculating Confidence Intervals for Cross Validated Binary Classifiers

I'm experimenting with a number of models for a binary classification problem. To evaluate the performance of each model, I've used 10x repeated 10-fold cross validation to calculate the PR AUC (Area Under the Precision-Recall Curve), and a number of other metrics.

My professor would like me to report the confidence intervals for these metrics and the p-value of the conclusion that model X is better than model Y. This is standard practice in Statistical Machine Translation (SMT) experiments (his background), where they use bootstrapping to calculate these (my experiment is NLP-related, but not SMT). Since I have used repeated cross-validation to estimate each model's performance, I'm not sure how to calculate the confidence intervals or p-values.

This question is similar, where they suggest bootstrapping the resampled mean, but I'm not sure what this means. Is this the correct approach, and, if so, can anyone explain how to do that in more detail?

Summary: Whatever you do in order to calculate confidence intervals based on repeated CV, you need to take into account that there are several different sources of uncertainty.

Long version: Let me add my 2 ct with regard to repeated cross validation:

Repeated cross validation allows you to separate 2 sources of variance uncertainty in the test results.

1. variance due to model instability, i.e. variance in the predictions for the same case by different surrogate models (i.e. exchange a few training cases in the surrogate models' training sets) and
2. variance due to the finite (limited) number of independent test cases

Now typically cross validation assumes that the models are stable, i.e. variance 1. is negligible. This assumption you can easily check. If you find it non-negligible, you'd typically go back to training and try to stabilize your models before doing anything else.
Variance 2 depends heavily on the total number of independent cases = the total number of independent cases tested in each run of the cross validation (and is usually much worse if you insist on classification rather than staying with metric scores).

I'm pointing this out because repetitions of the cross validation can help estimating and reducing variance 1, but will not mitigate variance 2 - but under the standard assumptions for cross validation, variance 2 should be dominating.
I suspect that this the underlying cause for VanWinckelen's finding that "Repeated cross-validation should not be assumed to give much more precise estimates of a model’s predic- tive accuracy".

If, instead of characterizing the model you get using the data at hand, you try to find out whether one or the other algorithm would be better for similar applications, you have more unknown sources of uncertainty, see Bengio and Grandvalet: No unbiased estimator of the variance of k-fold cross-validation.

Disclaimer: I cannot say much on the area under those curves as for my applications the curves are often skewed and I rather need to take into account pairs of figures of merit such as sens and spec or PPV and NPV (or scoring rules that are analogous to them).

• Thank you for that explanation. That makes some things much more clear. – Nimrand Jul 17 '17 at 11:05

Since you are computing areas, note that the AUROC is just the concordance probability $c$ between predicted risks and observed binary outcomes. And so you are not needing to engage in classification. Instead your outcomes can be just predicted risks. There are methods for getting confidence intervals for $c$-indexes by computing a standard error for $c$. But $c$ (AUROC) is not sensitive enough for there to be good statistical power for comparing two models. For that use the gold standard likelihood ratio $\chi^2$ test (if the models are nested) or compute the difference in two proper probability accuracy scores. You might also look into AIC for informal comparisons of non-nested models.

Formal $\chi^2$ or $F$ tests are done using pre-specified models and do not involve any resampling such as cross-validation. When more complex situations are present and you use a resampling procedure to unbiasedly estimate an accuracy score, things are more complicated and we don't have all the theory worked out yet. An option is always there - to use the double bootstrap, which is computationally intensive.

• I should clarify that this is a supervised learning problem, so I must split my data into training and test data. Furthermore, what I'm comparing is models trained with slightly different algorithms and/or different hyperparameters, trying to determine which combination of training algorithm/hyperparameters produces a model with the most accurate predictions for this kind of data. I am not evaluating a pre-specified model so much as I am evaluating the process for building/training a model for this kind of data. In light of that, can you clarify your answer? – Nimrand Jul 15 '17 at 13:38
• For example, if its still relevant, can you clarify what you mean by "embedded"? This seems to be an overloaded term and I'm not sure if my models are embedded in the sense that you mean or not. Thanks. – Nimrand Jul 15 '17 at 13:38
• Rather than a split-sample approach, a comprehensive all-in-one bootstrap resampling algorithm would be a more precise way to validate the model. – Frank Harrell Jul 15 '17 at 14:30
• I'm not sure I understand. If there is any overlap in my test and training data, my model evaluation results will be overly optimistic. Indeed, this was a problem in my early experiments until I realized that there were duplicates in my data that I needed to remove before splitting the data. How is it that I do not need to split the data into non-overlapping training and test samples? – Nimrand Jul 15 '17 at 21:56
• The Efron-Gong optimism bootstrap capitalizes on this overlap to produce the most precise and nearly unbiased estimates of model performance. It estimates the difference in performance between bootstrap samples (duplicated observations cause super overfitting) and evaluating the bootstrap-fitted model on the original sample (regular overfitting). – Frank Harrell Jul 16 '17 at 12:22