ROC and multiROC analysis: how to calculate optimal cutpoint?

Question

I'm trying to understand how to compute the optimal cut-point for a ROC curve (the value at which the sensitivity and specificity are maximized). I'm using the dataset aSAH from the package pROC.

The outcome variable could be explained by two independent variables: s100b and ndka. Using the syntax of the Epi package, I've created two models:

library(pROC)
library(Epi)
ROC(form=outcome~s100b, data=aSAH)
ROC(form=outcome~ndka, data=aSAH)

The output is illustrated in the following two graphs:

In the first graph (s100b), the function says that the optimal cut-point is localized at the value corresponding to lr.eta=0.304. In the second graph (ndka) the optimal cut-point is localized at the corresponding value to lr.eta=0.335 (what is the meaning of lr.eta). My first question is:

• what is the corresponding s100b and ndka values for the lr.eta values indicated (what is the optimal cut-point in terms of s100b and ndka)?

SECOND QUESTION:

Now suppose I create a model taking into account both variable:

ROC(form=outcome~ndka+s100b, data=aSAH)

The graph obtained is:

I want to know what are the values of ndka AND s100b at which sensibility and specificity are maximized by the function. In other terms: what are the values of ndka and s100b at which we have Se=68.3% and Sp=76.4% (values derived from the graph)?

I suppose this second question is related to multiROC analysis, but the documentation of the Epi package does not explain how to calculate the optimal cutpoint for both variables used in the model.

My question appears very similar to this question from reasearchGate, which says in short:

The determination of cut-off score that represents a better trade-off between sensitivity and specificity of a measure is straightforward. However, for multivariate ROC curve analysis, I have noted that most of the researchers have focused on algorithms to determine the overall accuracy of a linear combination of several indicators (variables) in terms of AUC. [...]

However, these methods do not mention how to decide a combination of cut-off scores associated with the multiple indicators that gives the best diagnostic accuracy.

A possible solution is that proposed by Shultz in his paper, but from this article I'm not able to understand how to compute optimal cutpoint for a multivariate ROC curve.

Maybe the solution from the Epi package is not ideal, so any other helpful links will be appreciated.

Answer 1

To elaborate on Frank Harrell's answer, what the Epi package did was to fit a logistic regression, and make a ROC curve with outcome predictions of the following form:

$$ outcome = \frac {1}{1+e^{-(\beta_0 + \beta_1 s100b + \beta_2 ndka)}} $$

In your case, the fitted values are $\beta_0$ (intercept) = -2.379, $\beta_1$ (s100b) = 5.334 and $\beta_2$ (ndka) = 0.031. As you want your predicted outcome to be 0.312 (the "optimal" cutoff), you can then substitute this as (hope I didn't introduce errors here):

$$ 0.312 = \frac {1}{1+e^{-(-2.379 + 5.334 s100b + 0.031 ndka)}} $$ $$ 1.588214 = 5.334 s100b + 0.031 ndka $$ or: $$ s100b = \frac{1.588214 - 0.031 ndka}{5.334} $$

Any pair of (s100b, ndka) values that satisfy this equality is "optimal". Bad luck for you, there are an infinity of these pairs. For instance, (0.29, 1), (0, 51.2), etc. Even worse, most of them don't make any sense. What does the pair (-580, 10000) mean? Nothing!

In other words, you can't establish cut-offs on the inputs - you have to do it on the outputs, and that's the whole point of the model.

Answer 2

It is not appropriate to seek cutoffs on input variables, but instead only on the output (e.g., predicted risk from a multivariable model). That is because the cutoff for x1 would depend on the continuous value of x2. And seeking a cutpoint on $\hat{Y}$, to obtain optimum decisions, requires a utility/loss/cost function and this has nothing to do with ROC curves.

Answer 3

I would guess lr.eta is the linear predictor—the logit—from the fitted model, as $\eta$ is a commonly used symbol for it; or, if not, the probability from the fitted model. (Turns out it's the latter: see https://stackoverflow.com/a/38532555/1864816.) You can check the code in ROC. In any case you'll be able to calculate it from the model coefficients for any number of predictors. (Note that it won't be a cut-off for each predictor separately, but a function of all predictors.)

Your first sentence should say (as evidenced by the graphs) that you're looking for where the sum of sensitivity & specificity is maximized. But why is this "optimal"? Does a false positive result have the same import as a false negative result? See here.

Answer 4

You can find the threshold at which the true positive rate (tpr) intersects the true negative rate (tnr) this will be point at which the sum of the false positives and false negatives is a minimum.