52
$\begingroup$

I have the data of a test that could be used to distinguish normal and tumor cells. According to ROC curve it looks good for this purpose (area under curve is 0.9):

ROC curve

My questions are:

  1. How to determine cutoff point for this test and its confidence interval where readings should be judged as ambiguous?
  2. What is the best way to visualize this (using ggplot2)?

Graph is rendered using ROCR and ggplot2 packages:

#install.packages("ggplot2","ROCR","verification") #if not installed yet
library("ggplot2")
library("ROCR")
library("verification")
d <-read.csv2("data.csv", sep=";")
pred <- with(d,prediction(x,test))
perf <- performance(pred,"tpr", "fpr")
auc <-performance(pred, measure = "auc")@y.values[[1]]
rd <- data.frame(x=perf@x.values[[1]],y=perf@y.values[[1]])
p <- ggplot(rd,aes(x=x,y=y)) + geom_path(size=1)
p <- p + geom_segment(aes(x=0,y=0,xend=1,yend=1),colour="black",linetype= 2)
p <- p + geom_text(aes(x=1, y= 0, hjust=1, vjust=0, label=paste(sep = "", "AUC = ",round(auc,3) )),colour="black",size=4)
p <- p + scale_x_continuous(name= "False positive rate")
p <- p + scale_y_continuous(name= "True positive rate")
p <- p + opts(
            axis.text.x = theme_text(size = 10),
            axis.text.y = theme_text(size = 10),
            axis.title.x = theme_text(size = 12,face = "italic"),
            axis.title.y = theme_text(size = 12,face = "italic",angle=90),
            legend.position = "none",
            legend.title = theme_blank(),
            panel.background = theme_blank(),
            panel.grid.minor = theme_blank(), 
            panel.grid.major = theme_line(colour='grey'),
            plot.background = theme_blank()
            )
p

data.csv contains the following data:

x;group;order;test
56;Tumor;1;1
55;Tumor;1;1
52;Tumor;1;1
60;Tumor;1;1
54;Tumor;1;1
43;Tumor;1;1
52;Tumor;1;1
57;Tumor;1;1
50;Tumor;1;1
34;Tumor;1;1
24;Normal;2;0
34;Normal;2;0
22;Normal;2;0
32;Normal;2;0
25;Normal;2;0
23;Normal;2;0
23;Normal;2;0
19;Normal;2;0
56;Normal;2;0
44;Normal;2;0
$\endgroup$
32
$\begingroup$

Thanks to all who aswered this question. I agree that there could be no one correct answer and criteria greatly depend on the aims that stand behind of the certain diagnostic test.

Finally I had found an R package OptimalCutpoints dedicated exactly to finding cutoff point in such type of analysis. Actually there are several methods of determining cutoff point.

  • "CB" (cost-benefit method);
  • "MCT" (minimizes Misclassification Cost Term);
  • "MinValueSp" (a minimum value set for Specificity);
  • "MinValueSe" (a minimum value set for Sensitivity);
  • "RangeSp" (a range of values set for Specificity);
  • "RangeSe" (a range of values set for Sensitivity);
  • "ValueSp" (a value set for Specificity);
  • "ValueSe" (a value set for Sensitivity);
  • "MinValueSpSe" (a minimum value set for Specificity and Sensitivity);
  • "MaxSp" (maximizes Specificity);
  • "MaxSe" (maximizes Sensitivity);
  • "MaxSpSe" (maximizes Sensitivity and Specificity simultaneously);
  • "Max-SumSpSe" (maximizes the sum of Sensitivity and Specificity);
  • "MaxProdSpSe" (maximizes the product of Sensitivity and Specificity);
  • "ROC01" (minimizes distance between ROC plot and point (0,1));
  • "SpEqualSe" (Sensitivity = Specificity);
  • "Youden" (Youden Index);
  • "MaxEfficiency" (maximizes Efficiency or Accuracy);
  • "Minimax" (minimizes the most frequent error);
  • "AUC" (maximizes concordance which is a function of AUC);
  • "MaxDOR" (maximizes Diagnostic Odds Ratio);
  • "MaxKappa" (maximizes Kappa Index);
  • "MaxAccuracyArea" (maximizes Accuracy Area);
  • "MinErrorRate" (minimizes Error Rate);
  • "MinValueNPV" (a minimum value set for Negative Predictive Value);
  • "MinValuePPV" (a minimum value set for Positive Predictive Value);
  • "MinValueNPVPPV" (a minimum value set for Predictive Values);
  • "PROC01" (minimizes distance between PROC plot and point (0,1));
  • "NPVEqualPPV" (Negative Predictive Value = Positive Predictive Value);
  • "ValueDLR.Negative" (a value set for Negative Diagnostic Likelihood Ratio);
  • "ValueDLR.Positive" (a value set for Positive Diagnostic Likelihood Ratio);
  • "MinPvalue" (minimizes p-value associated with the statistical Chi-squared test which measures the association between the marker and the binary result obtained on using the cutpoint);
  • "ObservedPrev" (The closest value to observed prevalence);
  • "MeanPrev" (The closest value to the mean of the diagnostic test values);
  • "PrevalenceMatching" (The value for which predicted prevalence is practically equal to observed prevalence).

So now the task is narrowed to selecting the method that is the best match for each situation.

There are many other configuration options described in package documentation including several methods of determining confidence intervals and detailed description of each of the methods.

$\endgroup$
  • 19
    $\begingroup$ The sheer number of methods is a sign of the arbitrariness of a cutoff. And since it is wholly inappropriate to use cutoffs on input variables, and only appropriate to seek at cutoff (if you must) on an overall predicted value, it is not clear why so much effort is spent on this. If you set up a Bayes optimum decision rule with a loss function everything gets taken care of; no ROC curve, no backwards-time probabilities such as sensitivity and specificity, no cutoffs on input variables. $\endgroup$ – Frank Harrell Aug 15 '13 at 12:07
  • $\begingroup$ @FrankHarrell Could you elaborate on this? "If you set up a Bayes optimum decision rule with a loss function everything gets taken care of." Where can I find more literature on this? $\endgroup$ – Black Milk Aug 7 '14 at 23:06
  • 1
    $\begingroup$ Look at the literature on Bayes optimal decisions and on proper scoring rules. $\endgroup$ – Frank Harrell Aug 8 '14 at 12:14
26
$\begingroup$

In my opinion, there are multiple cut-off options. You might weight sensitivity and specificity differently (for example, maybe for you it is more important to have a high sensitive test even though this means having a low specific test. Or vice-versa).

If sensitivity and specificity have the same importance to you, one way of calculating the cut-off is choosing that value that minimizes the Euclidean distance between your ROC curve and the upper left corner of your graph.

Another way is using the value that maximizes (sensitivity + specificity - 1) as a cut-off.

Unfortunately, I do not have references for these two methods as I have learned them from professors or other statisticians. I have only heard referring to the latter method as the 'Youden's index' [1]).

[1] https://en.wikipedia.org/wiki/Youden%27s_J_statistic

$\endgroup$
15
$\begingroup$

Resist the temptation to find a cutoff. Unless you have a pre-specified utility/loss/cost function, a cutoff flies in the face of optimal decision-making. And an ROC curve is irrelevant to this issue.

$\endgroup$
7
$\begingroup$

Mathematically speaking, you need another condition to solve for the cut-off.

You may translate @Andrea's point to: "use external knowledge about the underlying problem".

Example conditions:

  • for this application, we need sensitivity >= x, and/or specificity >= y.

  • a false negative is 10 x as bad as a false positive. (That would give you a modification of the closest point to the ideal corner.)

$\endgroup$
  • 1
    $\begingroup$ Exactly right that you need external knowledge to get the optimum decision. But the loss function is not stated in terms of the quantities above, and the optimum decision comes from the predicted probability of the outcome for the individual subject, coupled with the loss function. $\endgroup$ – Frank Harrell Dec 21 '15 at 14:04
6
$\begingroup$

Visualize accuracy versus cutoff. You can read more details at ROCR documentation and very nice presentation from the same.

enter image description here

$\endgroup$
  • 1
    $\begingroup$ If you look closer at source code I had used this package and read the documentation to this package. It has no tools to determine the right cutoff points and "grey zone" $\endgroup$ – Yuriy Petrovskiy Jul 8 '12 at 8:47
  • 1
    $\begingroup$ I definitely read your code but there is no such term as "right cutoff" but the plot Accuracy vs cutoff can give you the correct insight. And using this plot you can figure out how to find cutoff for maximum accuracy. $\endgroup$ – Vladimir Chupakhin Jul 8 '12 at 14:01
  • $\begingroup$ @Accuracy might be misleading when we've unbalanced data [accuracy pparadox]. $\endgroup$ – Dr Nisha Arora Nov 28 '19 at 9:42
2
$\begingroup$

What's more important - there's very few datapoints behind this curve. When you do decide how you're going to make the sensitivity/specificity tradeoff I'd strongly encourage you to bootstrap the curve and the resulting cutoff number. You may find that there's a lot of uncertainty in your estimated best cutoff.

$\endgroup$
  • 1
    $\begingroup$ The experimen is still under way, so I will get more data points. I am interested in methodology (I think it is the same for any count of data points). And I had not found any statistical method of determining "gray zone" while it is widely used in tests of such type. $\endgroup$ – Yuriy Petrovskiy Jun 4 '12 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.