5
$\begingroup$

I have an empirical estimate of a ROC curve, that is, a plot of the sensitivity versus 1-specificity over all possible cut-off values of the marker. Based on an empirical ROC curve, I would like to determine the optimal cut-off point that represents a better trade-off between sensitivity and specificity. I have read that the Youden Index can be used in that purpose.

Here is an example:

oneMinusSpecificity <- c(1.00000000, 
                         0.636363636,
                         0.436363636,
                         0.315151515,
                         0.163636364,
                         0.096969697,
                         0.072727273,
                         0.006060606,
                         0.000000000)
sensitivity <- c(1.00000000,
                 0.91566265,
                 0.77108434,
                 0.66265060,
                 0.39759036,
                 0.33734940,
                 0.28915663,
                 0.07228916,
                 0.00000000)

which results in the following ROC curve. The vertical line represents the Youden index = largest distance between "1 - specificity" and "sensitivity".

enter image description here

I find

> youden <- max(sensitivity - oneMinusSpecificity)
> youden
[1] 0.3474991

How can I calculate the optimal cut-off using this information?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
10
$\begingroup$

It is a mistake to think that an optimum threshold can be computed without knowing the cost of a false positive and the cost of a false negative for a specific subject. And if those costs are not identical for all subjects, it is easy to see that no threshold should be used. ROC curves and Youden indexes are only useful for mass one-time group decision making where utilities are unknowable. You are making a series of very subtle assumptions. One of these is that the binary choice is forced, i.e., there is no gray zone that would lead to a "defer the decision, get more data" action.

Improve this answer
$\endgroup$
6
  • $\begingroup$ @Harrell: Thanks for your comment. Here, I want to give as much importance to sensitivity as to specificity. $\endgroup$
    – user7064
    Commented May 13, 2013 at 15:11
  • $\begingroup$ Sensitivity and specificity have nothing to do with optimum decision making. In addition they are improper scoring rules, i.e., they are optimized by a bogus model. $\endgroup$
    – Frank Harrell
    Commented May 13, 2013 at 15:11
  • $\begingroup$ If I get it right, you do not agree with my strategy. What strategy would you recommend to find to optimal threshold? $\endgroup$
    – user7064
    Commented May 13, 2013 at 15:16
  • 1
    $\begingroup$ The desire to find a threshold is where things started getting messy. The main job of a statistical model is to estimate something, such as risk. Maximum likelihood estimation is your friend. Once you have a strongly validated risk model that shows itself to be well calibrated, you can use that model for decision making, because the utilities (costs) are seldom known until the model is applied. Then optimum decisions can be made for individual subjects, taking into account one subject's utility function. $\endgroup$
    – Frank Harrell
    Commented May 13, 2013 at 15:19
  • 1
    $\begingroup$ Just say "no". It's an invalid request. $\endgroup$
    – Frank Harrell
    Commented Apr 17, 2020 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.