Confidence intervals on a cutpoint of ROC curve

Can one estimate and calculate a confidence interval for the value of a cutpoint obtained from the ROC curve?

for example, using the pROC package in R:

> library(pROC)
> data(aSAH)
> roc1 <- roc(aSAH$outcome, aSAH$s100b, percent=TRUE,
# arguments for ci
ci=TRUE, boot.n=100, ci.alpha=0.9, stratified=FALSE,
# arguments for plot
plot=TRUE, auc.polygon=TRUE, max.auc.polygon=TRUE, grid=TRUE,
print.auc=TRUE, show.thres=TRUE)


with confidence intervals:

> ci.thresholds(roc1)


will produce:

95% CI (2000 stratified bootstrap replicates):
thresholds  sp.low sp.median sp.high se.low se.median se.high
-Inf   0.000      0.00    0.00 100.00    100.00  100.00
0.065   6.944     13.89   22.22  92.68     97.56  100.00
0.075  12.500     22.22   31.94  80.49     90.24   97.56
0.085  20.830     30.56   41.67  77.99     87.80   97.56
0.095  27.780     38.89   50.00  70.73     82.93   92.68
0.105  37.500     48.61   59.72  65.85     78.05   90.24
0.115  43.060     54.17   65.28  60.98     75.61   87.80
0.135  47.220     58.33   69.44  53.66     68.29   80.49
0.155  58.330     69.44   80.56  51.22     65.85   80.49
0.205  70.830     80.56   88.89  48.78     63.41   78.05
0.245  73.580     81.94   90.28  43.90     58.54   73.17
0.290  73.610     83.33   91.67  34.15     51.22   65.85
0.325  76.350     84.72   93.06  29.27     46.34   60.98
0.345  79.170     87.50   94.44  29.27     43.90   58.54
0.395  80.560     88.89   95.83  26.83     41.46   56.10
0.435  83.330     90.28   95.87  24.39     39.02   53.66
0.475  90.280     95.83  100.00  19.51     34.15   48.78
0.485  93.060     97.22  100.00  17.07     31.71   46.34
0.510 100.000    100.00  100.00  14.63     29.27   43.90


QUESTION

Why there is no CI on thresholds?

UPDATE

I realised how to specify the best cutpoint to be not youden, but topleft?

rocobj <- plot.roc(aSAH$outcome, aSAH$s100b,
main="Confidence intervals",
percent=TRUE,  ci=TRUE, print.auc=TRUE)
# print the AUC (will contain the CI)
ciobj <- ci.se(rocobj,
specificities=seq(0, 100, 5))
plot(ciobj, type="shape", col="#1c61b6AA")
plot(ci(rocobj, of="thresholds", thresholds="best", best.method="topleft"))

• Yes it's possible to inspect the sampling variability of a threshold that achieves a specific sensitivity (WLOG). Since the sensitivity has a relationship with a survival curve against the possible cutpoints, one can derive an expression for a, say 80% sensitive threshold CI. The bootstrap is probably an okay way to do this. Commented Apr 11, 2018 at 20:30

You fixed the thresholds. They cannot vary in the boostrap.

Let's simplify and look at only one threshold:

> ci.thresholds(roc1, thresholds = 0.205)
95% CI (2000 stratified bootstrap replicates):
thresholds sp.low sp.median sp.high se.low se.median se.high
0.205  70.83     80.56   88.89  48.78     63.41   78.05


By doing that, you fixed the threshold to 0.205 and asked: how much can my sensibility and specificity vary at that threshold? The threshold is your fixed point. It has no uncertainty associated with it. You implicitly did the same by asking for all thresholds, even though you didn't spell it out.

If you want a CI on the threshold you have to reformulate the question, for instance how uncertain is the "best" threshold?

> ci.coords(roc1, x="best", ret="threshold")
95% CI (2000 stratified bootstrap replicates):
2.5%   50% 97.5%
threshold best: threshold 0.155 0.205 0.485


Now of course you could ask: let's now fix the sensitivity at X = 0.9, and see how the threshold varies there. But this is more tricky than it sounds: you see, most of the ROC curve is actually a line between discrete points. Most points on the ROC curve actually fall between two thresholds. To calculate a threshold for an arbitrary point, you would need to interpolate a threshold value there. This is doable, but requires some parametric assumptions about the distribution of thresholds. Are you doing a linear interpolation? That would be pretty bad for most datasets I've ever seen.

So far, pROC is not able to reliably calculate a threshold at an arbitrary coordinate, so questions like how uncertain is the threshold at a fixed sensitivity X? result in errors or missing values. It's a can of worms I tried to open but I never reached anything useful.

• First of all, many thanks for your brilliant pROC package, it is extremely helpful (and also pedagogical!). I'm new to ROC analysis so need to dig deeper into this. On the same note, when you say "best" cutpoint, it certainly depends on the clinical criteria (Max Se, Sp, PPV, LDR, Youden....). Is there a way to choose the criteria in pROC to estimate the CI of a threshold? Commented Mar 27, 2018 at 23:03
• Another question: it appears, that it is impossible with pROC to estimate thresholds on the smoothed curved. In my case, I have only 18 instances in class 0 and 12 in class 1 (disease/non-disease). Is there a good way to oversample my instances to approximate quite rough real thresholds? Commented Mar 27, 2018 at 23:11
• @ArnoldKlein you can set the cost and prevalence with best.weights, such as plot(ci(rocobj, of="thresholds", thresholds="best", best.method="topleft", best.weights=c(10, 0.6))). See ?coords for more details. Commented Mar 28, 2018 at 14:28