3
$\begingroup$

I have an empirical estimate of a ROC curve, that is, a plot of the sensitivity versus 1-specificity over all possible cut-off values of the marker. Based on an empirical ROC curve, I would like to determine the optimal cut-off point that represents a better trade-off between sensitivity and specificity. I have read that the Youden Index can be used in that purpose.

Here is an example:

oneMinusSpecificity <- c(1.00000000, 
                         0.636363636,
                         0.436363636,
                         0.315151515,
                         0.163636364,
                         0.096969697,
                         0.072727273,
                         0.006060606,
                         0.000000000)
sensitivity <- c(1.00000000,
                 0.91566265,
                 0.77108434,
                 0.66265060,
                 0.39759036,
                 0.33734940,
                 0.28915663,
                 0.07228916,
                 0.00000000)

which results in the following ROC curve. The vertical line represents the Youden index = largest distance between "1 - specificity" and "sensitivity".

enter image description here

I find

> youden <- max(sensitivity - oneMinusSpecificity)
> youden
[1] 0.3474991

How can I calculate the optimal cut-off using this information?

$\endgroup$
  • $\begingroup$ Please note that you are also in a gray zone of this site's scope; this seems like an R programming Q which might be asked on StackOverflow. Anyway, it would be easier to answer if you would make it reproducible, i.e. add information how you have calculated sensitivity and oneMinusSpecificity from the decision and raw biomarker level. $\endgroup$ – user88 May 13 '13 at 15:52
5
$\begingroup$

It is a mistake to think that an optimum threshold can be computed without knowing the cost of a false positive and the cost of a false negative for a specific subject. And if those costs are not identical for all subjects, it is easy to see that no threshold should be used. ROC curves and Youden indexes are only useful for mass one-time group decision making where utilities are unknowable. You are making a series of very subtle assumptions. One of these is that the binary choice is forced, i.e., there is no gray zone that would lead to a "defer the decision, get more data" action.

$\endgroup$
  • $\begingroup$ @Harrell: Thanks for your comment. Here, I want to give as much importance to sensitivity as to specificity. $\endgroup$ – user7064 May 13 '13 at 15:11
  • $\begingroup$ Sensitivity and specificity have nothing to do with optimum decision making. In addition they are improper scoring rules, i.e., they are optimized by a bogus model. $\endgroup$ – Frank Harrell May 13 '13 at 15:11
  • $\begingroup$ If I get it right, you do not agree with my strategy. What strategy would you recommend to find to optimal threshold? $\endgroup$ – user7064 May 13 '13 at 15:16
  • 1
    $\begingroup$ The desire to find a threshold is where things started getting messy. The main job of a statistical model is to estimate something, such as risk. Maximum likelihood estimation is your friend. Once you have a strongly validated risk model that shows itself to be well calibrated, you can use that model for decision making, because the utilities (costs) are seldom known until the model is applied. Then optimum decisions can be made for individual subjects, taking into account one subject's utility function. $\endgroup$ – Frank Harrell May 13 '13 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.