14
$\begingroup$

I'm trying to understand how to compute the optimal cut-point for a ROC curve (the value at which the sensitivity and specificity are maximized). I'm using the dataset aSAH from the package pROC.

The outcome variable could be explained by two independent variables: s100b and ndka. Using the syntax of the Epi package, I've created two models:

library(pROC)
library(Epi)
ROC(form=outcome~s100b, data=aSAH)
ROC(form=outcome~ndka, data=aSAH)

The output is illustrated in the following two graphs:

enter image description here enter image description here

In the first graph (s100b), the function says that the optimal cut-point is localized at the value corresponding to lr.eta=0.304. In the second graph (ndka) the optimal cut-point is localized at the corresponding value to lr.eta=0.335 (what is the meaning of lr.eta). My first question is:

  • what is the corresponding s100b and ndka values for the lr.eta values indicated (what is the optimal cut-point in terms of s100b and ndka)?

SECOND QUESTION:

Now suppose I create a model taking into account both variable:

ROC(form=outcome~ndka+s100b, data=aSAH)

The graph obtained is:

enter image description here

I want to know what are the values of ndka AND s100b at which sensibility and specificity are maximized by the function. In other terms: what are the values of ndka and s100b at which we have Se=68.3% and Sp=76.4% (values derived from the graph)?

I suppose this second question is related to multiROC analysis, but the documentation of the Epi package does not explain how to calculate the optimal cutpoint for both variables used in the model.

My question appears very similar to this question from reasearchGate, which says in short:

The determination of cut-off score that represents a better trade-off between sensitivity and specificity of a measure is straightforward. However, for multivariate ROC curve analysis, I have noted that most of the researchers have focused on algorithms to determine the overall accuracy of a linear combination of several indicators (variables) in terms of AUC. [...]

However, these methods do not mention how to decide a combination of cut-off scores associated with the multiple indicators that gives the best diagnostic accuracy.

A possible solution is that proposed by Shultz in his paper, but from this article I'm not able to understand how to compute optimal cutpoint for a multivariate ROC curve.

Maybe the solution from the Epi package is not ideal, so any other helpful links will be appreciated.

$\endgroup$

4 Answers 4

10
$\begingroup$

To elaborate on Frank Harrell's answer, what the Epi package did was to fit a logistic regression, and make a ROC curve with outcome predictions of the following form:

$$ outcome = \frac {1}{1+e^{-(\beta_0 + \beta_1 s100b + \beta_2 ndka)}} $$

In your case, the fitted values are $\beta_0$ (intercept) = -2.379, $\beta_1$ (s100b) = 5.334 and $\beta_2$ (ndka) = 0.031. As you want your predicted outcome to be 0.312 (the "optimal" cutoff), you can then substitute this as (hope I didn't introduce errors here):

$$ 0.312 = \frac {1}{1+e^{-(-2.379 + 5.334 s100b + 0.031 ndka)}} $$ $$ 1.588214 = 5.334 s100b + 0.031 ndka $$ or: $$ s100b = \frac{1.588214 - 0.031 ndka}{5.334} $$

Any pair of (s100b, ndka) values that satisfy this equality is "optimal". Bad luck for you, there are an infinity of these pairs. For instance, (0.29, 1), (0, 51.2), etc. Even worse, most of them don't make any sense. What does the pair (-580, 10000) mean? Nothing!

In other words, you can't establish cut-offs on the inputs - you have to do it on the outputs, and that's the whole point of the model.

$\endgroup$
1
  • $\begingroup$ The logical follow-up question is: if we were faced with the task of considering a cutoff point based on the lr.eta in a model with only one explanatory variable, say s100b, would it be OK to select a cut-off point based on the lr.eta, excepting medical considerations regarding cost assigned to missed diagnoses, unnecessary procedures, etc? $\endgroup$ Commented Jul 14, 2020 at 2:44
8
$\begingroup$

It is not appropriate to seek cutoffs on input variables, but instead only on the output (e.g., predicted risk from a multivariable model). That is because the cutoff for x1 would depend on the continuous value of x2. And seeking a cutpoint on $\hat{Y}$, to obtain optimum decisions, requires a utility/loss/cost function and this has nothing to do with ROC curves.

$\endgroup$
7
  • $\begingroup$ I understand the problem you explained. I'm wondering, by the way, if there is a method to calculate cut-off points for two (or more) tests in parallel, in order to increase the Sens and Spec of identifying a specific status (disease/outcome/etc.). Thanks in advance. $\endgroup$
    – Tommaso
    Commented Aug 18, 2013 at 7:24
  • 1
    $\begingroup$ Since the "optimum" cutpoint for x1 would depend on the continuous value of x2, and the "optimum" cutpoint for x2 would depend on the continuous value of x1, there is no way to do this and preserve enough information to make it not a disaster. $\endgroup$ Commented Aug 18, 2013 at 11:44
  • $\begingroup$ So there is no way to find cut-off points for two or more tests, in order to maximize Sensibility and Specificity? Of course a method which is not a multiROC analysis. Thanks again. $\endgroup$
    – Tommaso
    Commented Sep 7, 2013 at 7:49
  • 2
    $\begingroup$ It is simply not appropriate to seek cutoffs on inputs. Optimum decisions are made using no cutoffs at all, or, if needed before decision time, by making cutoffs on the predicted probabilities. Utilities (loss/cost) are needed to solve for the optimum cutoff on predicted risk. $\endgroup$ Commented Sep 7, 2013 at 11:52
  • 1
    $\begingroup$ ROC curves have nothing whatsoever to do with meeting that goal. To do that you'd need to relate SCr to outcome, or just compute the probability of getting a more extreme SCr than that in a normal population. $\endgroup$ Commented Feb 19, 2016 at 19:17
3
$\begingroup$

I would guess lr.eta is the linear predictor—the logit—from the fitted model, as $\eta$ is a commonly used symbol for it; or, if not, the probability from the fitted model. (Turns out it's the latter: see https://stackoverflow.com/a/38532555/1864816.) You can check the code in ROC. In any case you'll be able to calculate it from the model coefficients for any number of predictors. (Note that it won't be a cut-off for each predictor separately, but a function of all predictors.)

Your first sentence should say (as evidenced by the graphs) that you're looking for where the sum of sensitivity & specificity is maximized. But why is this "optimal"? Does a false positive result have the same import as a false negative result? See here.

$\endgroup$
5
  • $\begingroup$ It was correct, I can calculate cut-off point from the fitted model (for one independent variable), or alternatively using the coords function from the pROC package, as I found later. The optimal cut-off point was, in my case, the best combination of Sens and Spec; I read the linked answer, but I don't care (at least for now) about false positive and false negative results, because (if I understood well) I'm analyzing a group of collected data for research. $\endgroup$
    – Tommaso
    Commented Aug 18, 2013 at 7:16
  • $\begingroup$ What do you care about then? What are you doing with the cut-off that doesn't require any consideration of consequences? And then what is it 'optimal' or 'best' for? $\endgroup$ Commented Aug 19, 2013 at 10:33
  • $\begingroup$ Tommaso defined "optimal" as "the value at which the sensitivity and specificity are maximized" (quoting the first sentence of the question), implicitly meaning max(sensitivity + specificity). Whether it makes sense or not (and when I read he doesn't care, I am inclined to think it does not) is an other question. $\endgroup$
    – Calimo
    Commented Feb 4, 2014 at 16:34
  • 1
    $\begingroup$ This approach is at odds with decision making. $\endgroup$ Commented Aug 18, 2015 at 12:28
  • 1
    $\begingroup$ I think that if I am reading your post correctly, lr.eta is exactly the second option that you mention: the probability from the fitted model: $E[Yi|Xi]=11+e−(β0+β1×s100b)$. Check this out if you have a minute. $\endgroup$ Commented Jul 22, 2016 at 18:00
0
$\begingroup$

You can find the threshold at which the true positive rate (tpr) intersects the true negative rate (tnr) this will be point at which the sum of the false positives and false negatives is a minimum.

$\endgroup$
2
  • $\begingroup$ A one-sentence answer is typically regarded as a little short for our format. Can you expand your answer to include a brief explanation of how you know that's where the minimum must be? $\endgroup$
    – Glen_b
    Commented Feb 24, 2015 at 4:04
  • 1
    $\begingroup$ Such a strategy flies in the face of optimum decision making. $\endgroup$ Commented Aug 18, 2015 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.