How to determine best cutoff point and it`s confidence interval using ROC curve in R?

Question

I have the data of a test that could be used to distinguish normal and tumor cells. According to ROC curve it looks good for this purpose (area under curve is 0.9):

My questions are:

1. How to determine cutoff point for this test and it`s confidence interval where readings should be judged as ambiguous?
2. What is the best way to visualize this (using ggplot2)?

Graph is rednered using ROCR and ggplot2 packages:

#install.packages("ggplot2","ROCR","verification") #if not installed yet
library("ggplot2")
library("ROCR")
library("verification")
d <-read.csv2("data.csv", sep=";")
pred <- with(d,prediction(x,test))
perf <- performance(pred,"tpr", "fpr")
auc <-performance(pred, measure = "auc")@y.values[[1]]
rd <- data.frame(x=perf@x.values[[1]],y=perf@y.values[[1]])
p <- ggplot(rd,aes(x=x,y=y)) + geom_path(size=1)
p <- p + geom_segment(aes(x=0,y=0,xend=1,yend=1),colour="black",linetype= 2)
p <- p + geom_text(aes(x=1, y= 0, hjust=1, vjust=0, label=paste(sep = "", "AUC = ",round(auc,3) )),colour="black",size=4)
p <- p + scale_x_continuous(name= "False positive rate")
p <- p + scale_y_continuous(name= "True positive rate")
p <- p + opts(
            axis.text.x = theme_text(size = 10),
            axis.text.y = theme_text(size = 10),
            axis.title.x = theme_text(size = 12,face = "italic"),
            axis.title.y = theme_text(size = 12,face = "italic",angle=90),
            legend.position = "none",
            legend.title = theme_blank(),
            panel.background = theme_blank(),
            panel.grid.minor = theme_blank(), 
            panel.grid.major = theme_line(colour='grey'),
            plot.background = theme_blank()
            )
p

data.csv contains the following data:

x;group;order;test
56;Tumor;1;1
55;Tumor;1;1
52;Tumor;1;1
60;Tumor;1;1
54;Tumor;1;1
43;Tumor;1;1
52;Tumor;1;1
57;Tumor;1;1
50;Tumor;1;1
34;Tumor;1;1
24;Normal;2;0
34;Normal;2;0
22;Normal;2;0
32;Normal;2;0
25;Normal;2;0
23;Normal;2;0
23;Normal;2;0
19;Normal;2;0
56;Normal;2;0
44;Normal;2;0

Answer 1

There is not one best cut-off in my opinion. You might weight differently the importance of sensitivity and specificity (for example, maybe you think it's more important to have a high sensitive test even though this means having a low specific one. Or vice-versa).

If they have the same "importance" to you, one way of calculating the cut-off is choosing that value that minimizes the euclidean distance between your ROC curve and the upper left corner of your graph.

Another way is choosing as cut-off that value that maximizes the sum of sensitivity and specificity.

(Unfortunately I don't have references for these two methods as I've learned them from professors or other statisticians doing their job. I've only heard referring to the first method as the "Jouden method").

Answer 2

Mathematically speaking, you need another condition to solve for the cut-off.

You may translate @Andrea's point to: "use external knowledge about the underlying problem".

Example conditions:

• for this application, we need sensitivity >= x, and/or specificity >= y.

• a false negative is 10 x as bad as a false positive. (That would give you a modification of the closest point to the ideal corner.)

Answer 3

What's more important - there's very few datapoints behind this curve. When you do decide how you're going to make the sensitivity/specificity tradeoff I'd strongly encourage you to bootstrap the curve and the resulting cutoff number. You may find that there's a lot of uncertainty in your estimated best cutoff.

Answer 4

Visualize accuracy versus cutoff. You can read more details at ROCR documentation and very nice presentation from the same.