Finding a cut-off point

Question

I'm trying to find a statistically significant cut-off point for number of attempts after which the risk of a poor outcome significantly increases.

I'm using SPSS. First I generated a ROC curve in SPSS which yielded an AUC of 0.649, and using the coordinates for that, 1.5 attempts (Sens=0.821, spec=0.494), rounded to 2, is the cut-off point.

So then, I generated a binary logistic regression model using multiple potential predictors of poor outcome (based on clinical knowledge), including the binary variables 2 or more attempts (y/n), 3 or more attempts (y/n), four or more attempts (y/n), and number of attempts (continuous). Using this method of analysis, four or more attempts as a categorical variable is the only significant predictor of poor outcome, but only if I include number of attempts as a continuous variable in the model. If I remove number of attempts, four or more attempts is no longer significant.

So, here are my questions:

1) How do I interpret this? Is the cut-off 2 or 4? Since the AUC is on the low side, is the ROC curve the best way to analyze the data?

2) Is it valid to have a logistic regression model with the same variable included in both continuous and binary form?

Thanks for your time.

Answer 1

First question: Finding "a statistically significant cut-off point" is much less important than finding a good probability model and combining it with knowledge of the subject matter. Your receiver operating characteristic (ROC) curve represents a whole set of potential tradeoffs between false positives and false negatives. That's a helpful display even with a low area under the curve (AUC); your AUC of 0.65 simply means that, for all pairs of comparable cases, the one you predicted to have a worse outcome actually did so 65% of the time.

Your particular choice of 2 attempts for a cutoff based on the ROC required some assumption about that sensitivity/specificity cutoff. There is a risk with statistical software that this assumption is implicit and hidden rather than explicit. For example, maybe it was chosen for you based on the farthest distance of the ROC curve from the diagonal line representing no discrimination. But does a sensitivity of 0.82 and a specificity of 0.49 really represent the best tradeoff from the perspective of the clinician who has to decide whether to continue trying? That depends on knowledge of the subject matter and shouldn't be left to an arbitrary choice by a statistical software routine.

Second question: You don't really have a continuous predictor, as the values are limited to integer numbers of trials. What you've done by also including binary predictors based on the number of attempts is to have imposed a particular general form for how the number of trials will be weighted.

Let's say that the regression coefficient for the "continuous" number of attempts is $\beta_1$, the coefficient for the binary representing 2 or more attempts is $\beta_2$, and so on. After you run your regression you get a linear predictor for each case based on the sum of the products of predictor values with their coefficients. The way you have coded the problem, the linear predictor values for the indicated number of attempts is:

1. $\beta_1$
2. $2\beta_1+\beta_2$
3. $3\beta_1+\beta_2+\beta_3$
4. $4\beta_1+\beta_2+\beta_3+\beta_4$
5. $5\beta_1+\beta_2+\beta_3+\beta_4$

That's not invalid, it's just represents another implicit assumption, in this case about the form of weighting different numbers of attempts.

You might consider treating the number of attempts as an ordinal predictor; answers on this page suggest several ways to proceed, with orthogonal polynomials or spline fits for the number of attempts. However you code the predictors, though, emphasize the overall quality of the probability model, and base any choice of cutoff on a clinically indicated rather than a "statistically signficant" tradeoff between specificity and sensitivity.

For example, in your logistic regression model don't use the statistical significance of the individual regression coefficients to choose the cutoff, but use the predicted probability to help inform the choice of whether to proceed. In practice, as much as clinicians might sometimes claim to want a cutoff, they typically include (sometimes unconsciously) lots of probabilities about various clinical aspects as they make their decisions in individual cases.

Alternate approach: It's not clear to me that your logistic regression model captures all that is going on here. Each trial seems to have 3 potential outcomes: success at unclogging, failure at unclogging, or an undesired result (injuring blood vessel, device breakage). After the first and last of these outcomes there are presumably no more attempts. If there is a simple failure to unclog, then the clinician decides whether to make another attempt.

So you don't know whether unclogging was impossible unless the clinician kept on trying until there was an undesired result. Thus your logistic regression might actually be modeling the probability of a clinician's giving up rather than an inherent property of clogged vessels.

You might instead consider modeling these probabilities of outcomes and the choice of whether to continue following a simple failure to unclog, as functions of the number of attempts. For example, are these probabilities constant, or do they change with the number of attempts? After how many attempts is the probability of unclogging at the next attempt offset by the risk (and costs) of an undesired outcome? That could be more useful than a logistic regression model.