I have to calculate the CI of the AUC (Roc) for a series of classifiers (e.g. Lasso, Random Forest, SVM) learned using the same test dataset, in order to identify the best model for this problem (prediction of a dichotomous variable).

Considering the small size of the dataset, I used the less known but almost unbiased Leave-Pair-Out-Cross-Validation (Airola et al.,2010). In brief, you learn and validate the model holding-out all possible combinations made of one case of one class and another case of the other class. Thus the (several) folds of the cross-validation procedure overlaps, with each case re-used in different validation folds.

The AUC is calculated according the Wilcoxon statistic (i.e as the average of all folds results, considering 1 if the p(C1) > p(C2) and 0 otherwise) as indicated in the papers where LPOCV was proposed. However I couldn't find in the literature any formal procedure to calculate the CI.

I thus decided to use a bootstrap procedure but I'm not sure that the design of the resampling procedure I applied is the correct one given the structure of LPOCV. 

I independently resampled (with repetition) the cases belonging to each of the two classes. Then I considered the folds according to the combination of two resamples, calculating the AUC as before. I performed these several times to obtain a distribution of bootstrapped AUC.

Is this correct? are there better resampling scheme to apply in this case?

Thank you! 

  • $\begingroup$ In addition to getting a Monte Carlo approximation to the bootstrap by applying your classification procedures to calculate area under the curve for the ROC and getting an approximate bootstrap distribution for AUC you have options for construction bootstrap confidence intervals. $\endgroup$ – Michael Chernick Dec 21 '16 at 20:13
  • $\begingroup$ Thank you @MichaelChernick. Can you please better explain me what you are proposing to do? (ps I used BCa CI) $\endgroup$ – Massimiliano Grassi Dec 22 '16 at 12:36
  • $\begingroup$ BCa is one of the better options. $\endgroup$ – Michael Chernick Jan 7 '17 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.