# Confidence Intervals for AUC using cross-validation

I am analyzing the performance of a predictive model with the AUC, area under the ROC curve. I repeat several times cross-validation, and I have different estimations of the AUC in each folder. For example, I repeat 10 times 10-fold CV and then, I have 100 estimations of AUC where I can calculate the MEAN(AUC) and the SD(AUC). My question is: how could I use this for calculate a 95% confidence interval for the AUC? These are some posible answers, but I am not sure if they are correct:

(1) Percentile 0.025 and 0.975 of the 100 sorted AUCs

(2) [ MEAN(AUC) - 1.96*SD(AUC) , MEAN(AUC) + 1.96*SD(AUC) ]

(3) [ MEAN(AUC) - 1.96*(SD(AUC)/sqrt(100)) , MEAN(AUC) + 1.96*(SD(AUC)/sqrt(100)) ]

Some comments: - The (3) is similar to (2) but taking into account the sample size determined by the number of repetitions I decide to do, and then, it will be narrow if I increase these repetitions - The intervals generated by (2) and (3) are symmetric

What do you think ? Thank

• 1: (1) is not a confidence interval. 2: It might be incorrect to assume that the 100 values are independently identically distributed from some stable distribution when there is clear dependence in the folding. 3: Is there some reason you want a confidence interval rather than just report (1) as a way to describe the 'range' of possibilities? – user44764 May 27 '14 at 12:20
• I am analysing the data with different predictive methods. I want to have a general idea to know if some methods is best, but I don't find a formal statistical test – Jesus Herranz Valera May 27 '14 at 12:30
• If you look at the 10 iterations of the 10-fold cross-validation, you capture exclusively variance due to instability of the surrogate models and do not observe the variance due to the fininte number of actual cases tested: each case is tested exactly once in each iteration. – cbeleites unhappy with SX May 28 '14 at 21:06
• Btw Steyerberg, Ewout W., et al. "Internal validation of predictive models: efficiency of some procedures for logistic regression analysis." Journal of clinical epidemiology 54.8 (2001): 774-781. calls it a "10x10% cross-validation". – Franck Dernoncourt Aug 2 '14 at 1:00

Here is a sample of how you would do it in python.

from sklearn import cross_validation
scores = cross_validation.cross_val_score(your_model, your_data, y, cv=10)
mean_score = scores.mean()
std_dev = scores.std()
std_error = scores.std() / math.sqrt(scores.shape[0])
ci =  2.262 * std_error
lower_bound = mean_score - ci
upper_bound = mean_score + ci

print "Score is %f +/-  %f" % (mean_score, ci)
print '95 percent probability that if this experiment were repeated over and
over the average score would be between %f and %f' % (lower_bound, upper_bound)

• This appears to be option (C), but with 2.262 instead of 1.96 (where does that come from?) and only one repetition of CV. – Danica Mar 13 '16 at 2:38
• 2.262 corresponds to the t-value for a sample size of 10 at the 95% confidence level stat.purdue.edu/~lfindsen/stat503/Table%2011%20-%20b.pdf – Benny Baysinger Mar 13 '16 at 6:36
• @Dougal I can see you have more experience in stats than I do. Do you disagree with the t-value? If so can you tell me why it isn't valid? – Benny Baysinger Apr 2 '16 at 19:53