# how to caculate 95% CI for AUC? try 384 times or Hanley et al. (1982) method?

I am working on a prediction task to predict heart disease risk. The data size is around 1500 and is splitted into train, validate and test datasets. I am use train dataset to train and use validate dataset to fine tune hyper parameters. Then, use test data and fine tuned model to caculate final AUC. The algorithm is random forest. I need to get the 95% CI of AUC, I find there are two ways to do that:

1, https://www.real-statistics.com/descriptive-statistics/roc-curve-classification-table/auc-confidence-interval/ this method only need do one classificaion test to get AUC. Then get 95% CI AUC by y_true, y_score.

2, Do prediction test 384 times (which can achieve 95% CI by simple random sampling), get mean and std(standard deviation) of 384 times AUC, and 95% CI AUC and be calculate by (mean - 1.96Xstd/19.60, mean + 1.96Xstd/19.60)

The data size is small, the AUC result depends on the split result, when I try 10 times I can get AUC interval widly from 0.6~0.9. And if I fix split random seed as 42, I can get AUC around 0.8. But, I think the first method is not a good way to get a reliable 95% CI AUC even I fix the random seed into 42. I think if the dataset is big enoungh and AUC results are stable, we should use the method1, but the dataset is too small and AUC results are not stable, we should use the method2, am I right? Which one do you think is the right way to get 95% CI AUC in my task? Thank you so much!

• You can't compute the c-index (AUROC) when doing a forced-choice classification task. You should label what you are doing as a prediction task. Classification means there are only 2 distinct outcomes (0 or 1) if Y is binary. Nov 19, 2021 at 13:35
• @FrankHarrell Thank you for your comment. Yes, I am doing binary (y = 0/1) classification. and y_true is the label of each data, looks like[0,1,1,1,1,0 ... ] and y_score is the probability score of predict as 1, looks like [0.45,0.01,0.78 ... ]. As I know, Forced-choice classification is binay classification, I have read a lot of paper such as predict the risk of Coronary Artery Disease, it is Forced-choice classification/binay classification and the paper have AUC and 95% CI AUC results. By the way, as I know, c-index is not AUROC. C-index is not c-statistics. Nov 29, 2021 at 8:51
• No. The c index = c statistic = auroc and you are NOT doing classification. You are doing prediction. Nov 29, 2021 at 14:11
• @ZzksZzks Keep in mind that there is a difference between the discrete measurement of your $y$ variable (binary category, either it is or is isn't) and the continuous probability prediction by the machine learning model.
– Dave
Nov 29, 2021 at 14:59
• @FrankHarrell Thank you for your comment! The concordance index or C-index is a generalization of the area under the ROC curve (AUC)! Thank you for pointing out my mistake. Nov 30, 2021 at 1:23