10
$\begingroup$

I am analyzing the performance of a predictive model with the AUC, area under the ROC curve. I repeat several times cross-validation, and I have different estimations of the AUC in each folder. For example, I repeat 10 times 10-fold CV and then, I have 100 estimations of AUC where I can calculate the MEAN(AUC) and the SD(AUC). My question is: how could I use this for calculate a 95% confidence interval for the AUC? These are some posible answers, but I am not sure if they are correct:

(1) Percentile 0.025 and 0.975 of the 100 sorted AUCs

(2) [ MEAN(AUC) - 1.96*SD(AUC) , MEAN(AUC) + 1.96*SD(AUC) ]

(3) [ MEAN(AUC) - 1.96*(SD(AUC)/sqrt(100)) , MEAN(AUC) + 1.96*(SD(AUC)/sqrt(100)) ]

Some comments: - The (3) is similar to (2) but taking into account the sample size determined by the number of repetitions I decide to do, and then, it will be narrow if I increase these repetitions - The intervals generated by (2) and (3) are symmetric

What do you think ? Thank

$\endgroup$
4
  • 2
    $\begingroup$ 1: (1) is not a confidence interval. 2: It might be incorrect to assume that the 100 values are independently identically distributed from some stable distribution when there is clear dependence in the folding. 3: Is there some reason you want a confidence interval rather than just report (1) as a way to describe the 'range' of possibilities? $\endgroup$ – user44764 May 27 '14 at 12:20
  • 1
    $\begingroup$ I am analysing the data with different predictive methods. I want to have a general idea to know if some methods is best, but I don't find a formal statistical test $\endgroup$ – Jesus Herranz Valera May 27 '14 at 12:30
  • 2
    $\begingroup$ If you look at the 10 iterations of the 10-fold cross-validation, you capture exclusively variance due to instability of the surrogate models and do not observe the variance due to the fininte number of actual cases tested: each case is tested exactly once in each iteration. $\endgroup$ – cbeleites unhappy with SX May 28 '14 at 21:06
  • 1
    $\begingroup$ Btw Steyerberg, Ewout W., et al. "Internal validation of predictive models: efficiency of some procedures for logistic regression analysis." Journal of clinical epidemiology 54.8 (2001): 774-781. calls it a "10x10% cross-validation". $\endgroup$ – Franck Dernoncourt Aug 2 '14 at 1:00
0
$\begingroup$

Here is a sample of how you would do it in python.

from sklearn import cross_validation
scores = cross_validation.cross_val_score(your_model, your_data, y, cv=10)
mean_score = scores.mean()
std_dev = scores.std()
std_error = scores.std() / math.sqrt(scores.shape[0])
ci =  2.262 * std_error
lower_bound = mean_score - ci
upper_bound = mean_score + ci

print "Score is %f +/-  %f" % (mean_score, ci)
print '95 percent probability that if this experiment were repeated over and    
over the average score would be between %f and %f' % (lower_bound, upper_bound)
$\endgroup$
3
  • 1
    $\begingroup$ This appears to be option (C), but with 2.262 instead of 1.96 (where does that come from?) and only one repetition of CV. $\endgroup$ – Danica Mar 13 '16 at 2:38
  • 2
    $\begingroup$ 2.262 corresponds to the t-value for a sample size of 10 at the 95% confidence level stat.purdue.edu/~lfindsen/stat503/Table%2011%20-%20b.pdf $\endgroup$ – Benny Baysinger Mar 13 '16 at 6:36
  • 2
    $\begingroup$ @Dougal I can see you have more experience in stats than I do. Do you disagree with the t-value? If so can you tell me why it isn't valid? $\endgroup$ – Benny Baysinger Apr 2 '16 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.