1
$\begingroup$

Background:

I have a dataset with about 2500 rows. I need to get the 95% confidence interval for my ROCs. I have trained this model on 80% of the data and 20% is for testing.

Where I am stuck:

Method 1

I am confused about how we get the CI for this classifier. From my understanding we can use a bootstrap method to obtain this. What is confusing for me is that some folk resample (with replacement) the dataset and fit the model to that resampled dataset. Then compute the ROC and get the AUC. Do this about n times and build a histogram of the AUCs as shown here, here, and here.

Method 2

I have seen others have trained a single model on the training data and then are tested using the test set to produce y_true and y_pred for the test set. We then sample the y_true and y_pred compute the ROC and get the AUC. Do this about n times and build a histogram of the AUCs as shown here, and here (Granted they use the pROC but only on single models y_test and y_pred)

Questions:

  1. What are the differences between the two?
  2. Why choose one over the other?
  3. What about a complex model that takes time to train and may be infeasible to do lots of iterations?
Improve this question
$\endgroup$
4
  • $\begingroup$ Do you want a confidence interval for the AUC or a confidence band for the ROC curve? Frank Harrell doubts the usefulness of the ROC curve itself. (He's a professor at Vanderbilt and a contributor on Cross Validated, and he's not a huge fan of AUC, either.) $\endgroup$
    – Dave
    Commented Aug 4, 2021 at 14:40
  • $\begingroup$ @Dave Its a good question I was asked to compute the confidence interval of the ROC. I assume this to be the CI of the AUC. But wouldn't the confidence band be found in a similar way (i.e bootstrapping)? $\endgroup$
    – Kevin
    Commented Aug 4, 2021 at 14:59
  • 1
    $\begingroup$ The AUC gives an indication of the classifiers ability to rank according to the propensity to belong to one class rather than another. If you are interested in ranking things, the AUC is a useful metric. If you need calibrated probabilities, there are better alternatives. I am always somewhat dubious of claims that techniques that have proven useful in the past are of no utility, it is usually a matter of the right tool for the job and it is a useful tool for some other task if it is of no utility for this one. $\endgroup$
    – Dikran Marsupial
    Commented Aug 4, 2021 at 15:07
  • 1
    $\begingroup$ Some additional posts on this topic can be found using search: stats.stackexchange.com/… A useful reference is ROC Curves for Continuous Data (Chapman & Hall/CRC Monographs on Statistics and Applied Probability) by Wojtek J. Krzanowski & David J. Hand. Note that asking for code is not an on-topic question on stats.SE. $\endgroup$
    – Sycorax ♦
    Commented Jan 31, 2022 at 16:39

0

Reset to default

Browse other questions tagged or ask your own question.