12
$\begingroup$

I'm trying to understand how to compute the optimal cut-point for a ROC curve (the value at which the sensitivity and specificity are maximized). I'm using the dataset aSAH from the package pROC.

The outcome variable could be explained by two independent variables: s100b and ndka. Using the syntax of the Epi package, I've created two models:

library(pROC)
library(Epi)
ROC(form=outcome~s100b, data=aSAH)
ROC(form=outcome~ndka, data=aSAH)

The output is illustrated in the following two graphs:

enter image description here enter image description here

In the first graph (s100b), the function says that the optimal cut-point is localized at the value corresponding to lr.eta=0.304. In the second graph (ndka) the optimal cut-point is localized at the corresponding value to lr.eta=0.335 (what is the meaning of lr.eta). My first question is:

  • what is the corresponding s100b and ndka values for the lr.eta values indicated (what is the optimal cut-point in terms of s100b and ndka)?

SECOND QUESTION:

Now suppose I create a model taking into account both variable:

ROC(form=outcome~ndka+s100b, data=aSAH)

The graph obtained is:

enter image description here

I want to know what are the values of ndka AND s100b at which sensibility and specificity are maximized by the function. In other terms: what are the values of ndka and s100b at which we have Se=68.3% and Sp=76.4% (values derived from the graph)?

I suppose this second question is related to multiROC analysis, but the documentation of the Epi package does not explain how to calculate the optimal cutpoint for both variables used in the model.

My question appears very similar to this question from reasearchGate, which says in short:

The determination of cut-off score that represents a better trade-off between sensitivity and specificity of a measure is straightforward. However, for multivariate ROC curve analysis, I have noted that most of the researchers have focused on algorithms to determine the overall accuracy of a linear combination of several indicators (variables) in terms of AUC. [...]

However, these methods do not mention how to decide a combination of cut-off scores associated with the multiple indicators that gives the best diagnostic accuracy.

A possible solution is that proposed by Shultz in his paper, but from this article I'm not able to understand how to compute optimal cutpoint for a multivariate ROC curve.

Maybe the solution from the Epi package is not ideal, so any other helpful links will be appreciated.

$\endgroup$
9
$\begingroup$

To elaborate on Frank Harrell's answer, what the Epi package did was to fit a logistic regression, and make a ROC curve with outcome predictions of the following form:

$$ outcome = \frac {1}{1+e^{-(\beta_0 + \beta_1 s100b + \beta_2 ndka)}} $$

In your case, the fitted values are $\beta_0$ (intercept) = -2.379, $\beta_1$ (s100b) = 5.334 and $\beta_2$ (ndka) = 0.031. As you want your predicted outcome to be 0.312 (the "optimal" cutoff), you can then substitute this as (hope I didn't introduce errors here):

$$ 0.312 = \frac {1}{1+e^{-(-2.379 + 5.334 s100b + 0.031 ndka)}} $$ $$ 1.588214 = 5.334 s100b + 0.031 ndka $$ or: $$ s100b = \frac{1.588214 - 0.031 ndka}{5.334} $$

Any pair of (s100b, ndka) values that satisfy this equality is "optimal". Bad luck for you, there are an infinity of these pairs. For instance, (0.29, 1), (0, 51.2), etc. Even worse, most of them don't make any sense. What does the pair (-580, 10000) mean? Nothing!

In other words, you can't establish cut-offs on the inputs - you have to do it on the outputs, and that's the whole point of the model.

$\endgroup$
8
$\begingroup$

It is not appropriate to seek cutoffs on input variables, but instead only on the output (e.g., predicted risk from a multivariable model). That is because the cutoff for x1 would depend on the continuous value of x2. And seeking a cutpoint on $\hat{Y}$, to obtain optimum decisions, requires a utility/loss/cost function and this has nothing to do with ROC curves.

$\endgroup$
  • $\begingroup$ I understand the problem you explained. I'm wondering, by the way, if there is a method to calculate cut-off points for two (or more) tests in parallel, in order to increase the Sens and Spec of identifying a specific status (disease/outcome/etc.). Thanks in advance. $\endgroup$ – Tommaso Aug 18 '13 at 7:24
  • 1
    $\begingroup$ Since the "optimum" cutpoint for x1 would depend on the continuous value of x2, and the "optimum" cutpoint for x2 would depend on the continuous value of x1, there is no way to do this and preserve enough information to make it not a disaster. $\endgroup$ – Frank Harrell Aug 18 '13 at 11:44
  • $\begingroup$ So there is no way to find cut-off points for two or more tests, in order to maximize Sensibility and Specificity? Of course a method which is not a multiROC analysis. Thanks again. $\endgroup$ – Tommaso Sep 7 '13 at 7:49
  • 2
    $\begingroup$ It is simply not appropriate to seek cutoffs on inputs. Optimum decisions are made using no cutoffs at all, or, if needed before decision time, by making cutoffs on the predicted probabilities. Utilities (loss/cost) are needed to solve for the optimum cutoff on predicted risk. $\endgroup$ – Frank Harrell Sep 7 '13 at 11:52
  • 1
    $\begingroup$ ROC curves have nothing whatsoever to do with meeting that goal. To do that you'd need to relate SCr to outcome, or just compute the probability of getting a more extreme SCr than that in a normal population. $\endgroup$ – Frank Harrell Feb 19 '16 at 19:17
3
$\begingroup$

I would guess lr.eta is the linear predictor—the logit—from the fitted model, as $\eta$ is a commonly used symbol for it; or, if not, the probability from the fitted model. (Turns out it's the latter: see https://stackoverflow.com/a/38532555/1864816.) You can check the code in ROC. In any case you'll be able to calculate it from the model coefficients for any number of predictors. (Note that it won't be a cut-off for each predictor separately, but a function of all predictors.)

Your first sentence should say (as evidenced by the graphs) that you're looking for where the sum of sensitivity & specificity is maximized. But why is this "optimal"? Does a false positive result have the same import as a false negative result? See here.

$\endgroup$
  • $\begingroup$ It was correct, I can calculate cut-off point from the fitted model (for one independent variable), or alternatively using the coords function from the pROC package, as I found later. The optimal cut-off point was, in my case, the best combination of Sens and Spec; I read the linked answer, but I don't care (at least for now) about false positive and false negative results, because (if I understood well) I'm analyzing a group of collected data for research. $\endgroup$ – Tommaso Aug 18 '13 at 7:16
  • $\begingroup$ What do you care about then? What are you doing with the cut-off that doesn't require any consideration of consequences? And then what is it 'optimal' or 'best' for? $\endgroup$ – Scortchi - Reinstate Monica Aug 19 '13 at 10:33
  • $\begingroup$ Tommaso defined "optimal" as "the value at which the sensitivity and specificity are maximized" (quoting the first sentence of the question), implicitly meaning max(sensitivity + specificity). Whether it makes sense or not (and when I read he doesn't care, I am inclined to think it does not) is an other question. $\endgroup$ – Calimo Feb 4 '14 at 16:34
  • 1
    $\begingroup$ This approach is at odds with decision making. $\endgroup$ – Frank Harrell Aug 18 '15 at 12:28
  • 1
    $\begingroup$ I think that if I am reading your post correctly, lr.eta is exactly the second option that you mention: the probability from the fitted model: $E[Yi|Xi]=11+e−(β0+β1×s100b)$. Check this out if you have a minute. $\endgroup$ – Antoni Parellada Jul 22 '16 at 18:00
0
$\begingroup$

You can find the threshold at which the true positive rate (tpr) intersects the true negative rate (tnr) this will be point at which the sum of the false positives and false negatives is a minimum.

$\endgroup$
  • $\begingroup$ A one-sentence answer is typically regarded as a little short for our format. Can you expand your answer to include a brief explanation of how you know that's where the minimum must be? $\endgroup$ – Glen_b -Reinstate Monica Feb 24 '15 at 4:04
  • 1
    $\begingroup$ Such a strategy flies in the face of optimum decision making. $\endgroup$ – Frank Harrell Aug 18 '15 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.