Confidence Intervals for AUC using cross-validation

Question

I am analyzing the performance of a predictive model with the AUC, area under the ROC curve. I repeat several times cross-validation, and I have different estimations of the AUC in each folder. For example, I repeat 10 times 10-fold CV and then, I have 100 estimations of AUC where I can calculate the MEAN(AUC) and the SD(AUC). My question is: how could I use this for calculate a 95% confidence interval for the AUC? These are some posible answers, but I am not sure if they are correct:

(1) Percentile 0.025 and 0.975 of the 100 sorted AUCs

(2) [ MEAN(AUC) - 1.96*SD(AUC) , MEAN(AUC) + 1.96*SD(AUC) ]

(3) [ MEAN(AUC) - 1.96*(SD(AUC)/sqrt(100)) , MEAN(AUC) + 1.96*(SD(AUC)/sqrt(100)) ]

Some comments: - The (3) is similar to (2) but taking into account the sample size determined by the number of repetitions I decide to do, and then, it will be narrow if I increase these repetitions - The intervals generated by (2) and (3) are symmetric

What do you think ? Thank

Answer 1

(1) and (2) are attempts to summarize the distribution of the estimates over repeating the cross-validation, but as @cbeleites unhappy with SX mentioned, this is not a meaningful interval for the true AUC. As mentioned by @user44764, your answer (3) is wrong as it tacitly assumes independence of AUC values across folds, which is wrong. It would only be correct if you had several AUC estimates of independent test datasets, and even then only apply to the specific training dataset, not to the AUC over all possible training datasets. To estimate the latter, you would need several sets of training and test datasets for which to calculate AUC, and then find the variance between them, which is rare. Instead, cross-validation is commonly used to estimate this latter AUC.

LeDell et al. (2015) provide an attractive method to find the confidence interval for the AUC, with R implementation: Computationally efficient confidence intervals for cross- validated area under the ROC curve estimates.

Answer 2

Here is a sample of how you would do it in python.

from sklearn import cross_validation
scores = cross_validation.cross_val_score(your_model, your_data, y, cv=10)
mean_score = scores.mean()
std_dev = scores.std()
std_error = scores.std() / math.sqrt(scores.shape[0])
ci =  2.262 * std_error
lower_bound = mean_score - ci
upper_bound = mean_score + ci

print "Score is %f +/-  %f" % (mean_score, ci)
print '95 percent probability that if this experiment were repeated over and    
over the average score would be between %f and %f' % (lower_bound, upper_bound)