0
$\begingroup$

I am working on a prediction task to predict heart disease risk. The data size is around 1500 and is splitted into train, validate and test datasets. I am use train dataset to train and use validate dataset to fine tune hyper parameters. Then, use test data and fine tuned model to caculate final AUC. The algorithm is random forest. I need to get the 95% CI of AUC, I find there are two ways to do that:

1, https://www.real-statistics.com/descriptive-statistics/roc-curve-classification-table/auc-confidence-interval/ this method only need do one classificaion test to get AUC. Then get 95% CI AUC by y_true, y_score.

2, Do prediction test 384 times (which can achieve 95% CI by simple random sampling), get mean and std(standard deviation) of 384 times AUC, and 95% CI AUC and be calculate by (mean - 1.96Xstd/19.60, mean + 1.96Xstd/19.60)

The data size is small, the AUC result depends on the split result, when I try 10 times I can get AUC interval widly from 0.6~0.9. And if I fix split random seed as 42, I can get AUC around 0.8. But, I think the first method is not a good way to get a reliable 95% CI AUC even I fix the random seed into 42. I think if the dataset is big enoungh and AUC results are stable, we should use the method1, but the dataset is too small and AUC results are not stable, we should use the method2, am I right? Which one do you think is the right way to get 95% CI AUC in my task? Thank you so much!

$\endgroup$
7
  • $\begingroup$ You can't compute the c-index (AUROC) when doing a forced-choice classification task. You should label what you are doing as a prediction task. Classification means there are only 2 distinct outcomes (0 or 1) if Y is binary. $\endgroup$ Nov 19, 2021 at 13:35
  • $\begingroup$ @FrankHarrell Thank you for your comment. Yes, I am doing binary (y = 0/1) classification. and y_true is the label of each data, looks like[0,1,1,1,1,0 ... ] and y_score is the probability score of predict as 1, looks like [0.45,0.01,0.78 ... ]. As I know, Forced-choice classification is binay classification, I have read a lot of paper such as predict the risk of Coronary Artery Disease, it is Forced-choice classification/binay classification and the paper have AUC and 95% CI AUC results. By the way, as I know, c-index is not AUROC. C-index is not c-statistics. $\endgroup$
    – Zzks Zzks
    Nov 29, 2021 at 8:51
  • $\begingroup$ No. The c index = c statistic = auroc and you are NOT doing classification. You are doing prediction. $\endgroup$ Nov 29, 2021 at 14:11
  • $\begingroup$ @ZzksZzks Keep in mind that there is a difference between the discrete measurement of your $y$ variable (binary category, either it is or is isn't) and the continuous probability prediction by the machine learning model. $\endgroup$
    – Dave
    Nov 29, 2021 at 14:59
  • $\begingroup$ @FrankHarrell Thank you for your comment! The concordance index or C-index is a generalization of the area under the ROC curve (AUC)! Thank you for pointing out my mistake. $\endgroup$
    – Zzks Zzks
    Nov 30, 2021 at 1:23

1 Answer 1

0
$\begingroup$

When constructing confidence intervals for complicated statistics, the task is simplified by using bootstrapping or Bayesian methods. For the former, try simulating data from your final fitted model and calculate the AUC a large number of times and apply bootstrapping CIs

$\endgroup$
4
  • $\begingroup$ Thank you very much for the answer! Do you mean I split the dataset 1 time, and use train and validate dataset to get final fitted model, then, use the test dataset and final fitted model to do bootstrapping ? $\endgroup$
    – Zzks Zzks
    Nov 29, 2021 at 8:36
  • $\begingroup$ Thank you very much for the answer! Do you mean I split the dataset 1 time, and use train and validate dataset to get the final fitted model, then, use the test dataset and final fitted model to do bootstrapping ? Is my understanding correct? But the AUC results are heavily depend on the data split result. I am afraid, with this method, I can only get 95%CI AUC of ONE time data split trial. $\endgroup$
    – Zzks Zzks
    Nov 29, 2021 at 8:51
  • $\begingroup$ Do you agree with this usage of bootstrap? stackoverflow.com/a/19132400/10893936 $\endgroup$
    – Zzks Zzks
    Nov 29, 2021 at 10:22
  • $\begingroup$ It says, To take the variability induced by the train test split into account, do random cross validation many times and create a new y_pred. $\endgroup$
    – Zzks Zzks
    Nov 29, 2021 at 10:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.