# Confidence Intervals for AUC using cross-validation

I am analyzing the performance of a predictive model with the AUC, area under the ROC curve. I repeat several times cross-validation, and I have different estimations of the AUC in each folder. For example, I repeat 10 times 10-fold CV and then, I have 100 estimations of AUC where I can calculate the MEAN(AUC) and the SD(AUC). My question is: how could I use this for calculate a 95% confidence interval for the AUC? These are some posible answers, but I am not sure if they are correct:

(1) Percentile 0.025 and 0.975 of the 100 sorted AUCs

(2) [ MEAN(AUC) - 1.96*SD(AUC) , MEAN(AUC) + 1.96*SD(AUC) ]

(3) [ MEAN(AUC) - 1.96*(SD(AUC)/sqrt(100)) , MEAN(AUC) + 1.96*(SD(AUC)/sqrt(100)) ]

Some comments: - The (3) is similar to (2) but taking into account the sample size determined by the number of repetitions I decide to do, and then, it will be narrow if I increase these repetitions - The intervals generated by (2) and (3) are symmetric

What do you think ? Thank

• 1: (1) is not a confidence interval. 2: It might be incorrect to assume that the 100 values are independently identically distributed from some stable distribution when there is clear dependence in the folding. 3: Is there some reason you want a confidence interval rather than just report (1) as a way to describe the 'range' of possibilities?
– user44764
May 27, 2014 at 12:20
• I am analysing the data with different predictive methods. I want to have a general idea to know if some methods is best, but I don't find a formal statistical test May 27, 2014 at 12:30
• If you look at the 10 iterations of the 10-fold cross-validation, you capture exclusively variance due to instability of the surrogate models and do not observe the variance due to the fininte number of actual cases tested: each case is tested exactly once in each iteration. May 28, 2014 at 21:06
• Btw Steyerberg, Ewout W., et al. "Internal validation of predictive models: efficiency of some procedures for logistic regression analysis." Journal of clinical epidemiology 54.8 (2001): 774-781. calls it a "10x10% cross-validation". Aug 2, 2014 at 1:00

(1) and (2) are attempts to summarize the distribution of the estimates over repeating the cross-validation, but as @cbeleites unhappy with SX mentioned, this is not a meaningful interval for the true AUC. As mentioned by @user44764, your answer (3) is wrong as it tacitly assumes independence of AUC values across folds, which is wrong. It would only be correct if you had several AUC estimates of independent test datasets, and even then only apply to the specific training dataset, not to the AUC over all possible training datasets. To estimate the latter, you would need several sets of training and test datasets for which to calculate AUC, and then find the variance between them, which is rare. Instead, cross-validation is commonly used to estimate this latter AUC.

LeDell et al. (2015) provide an attractive method to find the confidence interval for the AUC, with R implementation: Computationally efficient confidence intervals for cross- validated area under the ROC curve estimates.

Here is a sample of how you would do it in python.

from sklearn import cross_validation
scores = cross_validation.cross_val_score(your_model, your_data, y, cv=10)
mean_score = scores.mean()
std_dev = scores.std()
std_error = scores.std() / math.sqrt(scores.shape[0])
ci =  2.262 * std_error
lower_bound = mean_score - ci
upper_bound = mean_score + ci

print "Score is %f +/-  %f" % (mean_score, ci)
print '95 percent probability that if this experiment were repeated over and
over the average score would be between %f and %f' % (lower_bound, upper_bound)

• This appears to be option (C), but with 2.262 instead of 1.96 (where does that come from?) and only one repetition of CV. Mar 13, 2016 at 2:38
• 2.262 corresponds to the t-value for a sample size of 10 at the 95% confidence level stat.purdue.edu/~lfindsen/stat503/Table%2011%20-%20b.pdf Mar 13, 2016 at 6:36
• @Dougal I can see you have more experience in stats than I do. Do you disagree with the t-value? If so can you tell me why it isn't valid? Apr 2, 2016 at 19:53