Skip to main content
3 of 4
deleted 2 characters in body
Arnold Klein
  • 679
  • 1
  • 9
  • 18

Confidence intervals on a cutpoint of ROC curve

I am not sure whether my question is due to fundamental misunderstanding, but does it make sense to estimate a confidence interval (through bootstrap, DeLong, ...) on the value of a cutpoint obtained from the ROC curve?

for example, using the pROC package in R:

> library(pROC)
> data(aSAH)
> roc1 <- roc(aSAH$outcome,
            aSAH$s100b, percent=TRUE,
            # arguments for ci
            ci=TRUE, boot.n=100, ci.alpha=0.9, stratified=FALSE,
            # arguments for plot
            plot=TRUE, auc.polygon=TRUE, max.auc.polygon=TRUE, grid=TRUE,
            print.auc=TRUE, show.thres=TRUE)

with confidence intervals:

> ci.thresholds(roc1)

will produce:

95% CI (2000 stratified bootstrap replicates):
 thresholds  sp.low sp.median sp.high se.low se.median se.high
       -Inf   0.000      0.00    0.00 100.00    100.00  100.00
      0.065   6.944     13.89   22.22  92.68     97.56  100.00
      0.075  12.500     22.22   31.94  80.49     90.24   97.56
      0.085  20.830     30.56   41.67  77.99     87.80   97.56
      0.095  27.780     38.89   50.00  70.73     82.93   92.68
      0.105  37.500     48.61   59.72  65.85     78.05   90.24
      0.115  43.060     54.17   65.28  60.98     75.61   87.80
      0.135  47.220     58.33   69.44  53.66     68.29   80.49
      0.155  58.330     69.44   80.56  51.22     65.85   80.49
      0.205  70.830     80.56   88.89  48.78     63.41   78.05
      0.245  73.580     81.94   90.28  43.90     58.54   73.17
      0.290  73.610     83.33   91.67  34.15     51.22   65.85
      0.325  76.350     84.72   93.06  29.27     46.34   60.98
      0.345  79.170     87.50   94.44  29.27     43.90   58.54
      0.395  80.560     88.89   95.83  26.83     41.46   56.10
      0.435  83.330     90.28   95.87  24.39     39.02   53.66
      0.475  90.280     95.83  100.00  19.51     34.15   48.78
      0.485  93.060     97.22  100.00  17.07     31.71   46.34
      0.510 100.000    100.00  100.00  14.63     29.27   43.90

QUESTION

Why there is no CI on thresholds?

UPDATE

I realised how to specify the best cutpoint to be not youden, but topleft?

rocobj <- plot.roc(aSAH$outcome, 
                   aSAH$s100b,  
                   main="Confidence intervals", 
                   percent=TRUE,  ci=TRUE, print.auc=TRUE) 
# print the AUC (will contain the CI)  
ciobj <- ci.se(rocobj, 
               specificities=seq(0, 100, 5)) 
plot(ciobj, type="shape", col="#1c61b6AA")
plot(ci(rocobj, of="thresholds", thresholds="best", best.method="topleft")) 
Arnold Klein
  • 679
  • 1
  • 9
  • 18