0
$\begingroup$

I'm trying to find a statistically significant cut-off point for number of attempts after which the risk of a poor outcome significantly increases.

I'm using SPSS. First I generated a ROC curve in SPSS which yielded an AUC of 0.649, and using the coordinates for that, 1.5 attempts (Sens=0.821, spec=0.494), rounded to 2, is the cut-off point.

So then, I generated a binary logistic regression model using multiple potential predictors of poor outcome (based on clinical knowledge), including the binary variables 2 or more attempts (y/n), 3 or more attempts (y/n), four or more attempts (y/n), and number of attempts (continuous). Using this method of analysis, four or more attempts as a categorical variable is the only significant predictor of poor outcome, but only if I include number of attempts as a continuous variable in the model. If I remove number of attempts, four or more attempts is no longer significant.

So, here are my questions:

1) How do I interpret this? Is the cut-off 2 or 4? Since the AUC is on the low side, is the ROC curve the best way to analyze the data?

2) Is it valid to have a logistic regression model with the same variable included in both continuous and binary form?

Thanks for your time.

$\endgroup$
  • $\begingroup$ Could you please say more about why you need to define a cutoff and how that cutoff would be applied in practice? You seem already to have a model of the probability of a poor outcome, which many people on this site would consider preferable to an arbitrary cutoff. $\endgroup$ – EdM Nov 4 '18 at 14:58
  • $\begingroup$ Hi - it's mostly for clinical decision making. The scenario is this: a blood vessel is clogged, and you can try to unclog it with a device, but it may be that the clog isn't going to come out, regardless of how many times you put a device into it. Each time you put a device into it, you risk injuring the blood vessel and breaking the device. So, the question I'm trying to answer is "After how many unsuccessful attempts do my chances of having success significantly decrease?", thus the ROC curve with number of attempts as the test variable and failed unclogging as the state variable. $\endgroup$ – Statsdunce Nov 5 '18 at 23:20
2
$\begingroup$

First question: Finding "a statistically significant cut-off point" is much less important than finding a good probability model and combining it with knowledge of the subject matter. Your receiver operating characteristic (ROC) curve represents a whole set of potential tradeoffs between false positives and false negatives. That's a helpful display even with a low area under the curve (AUC); your AUC of 0.65 simply means that, for all pairs of comparable cases, the one you predicted to have a worse outcome actually did so 65% of the time.

Your particular choice of 2 attempts for a cutoff based on the ROC required some assumption about that sensitivity/specificity cutoff. There is a risk with statistical software that this assumption is implicit and hidden rather than explicit. For example, maybe it was chosen for you based on the farthest distance of the ROC curve from the diagonal line representing no discrimination. But does a sensitivity of 0.82 and a specificity of 0.49 really represent the best tradeoff from the perspective of the clinician who has to decide whether to continue trying? That depends on knowledge of the subject matter and shouldn't be left to an arbitrary choice by a statistical software routine.

Second question: You don't really have a continuous predictor, as the values are limited to integer numbers of trials. What you've done by also including binary predictors based on the number of attempts is to have imposed a particular general form for how the number of trials will be weighted.

Let's say that the regression coefficient for the "continuous" number of attempts is $\beta_1$, the coefficient for the binary representing 2 or more attempts is $\beta_2$, and so on. After you run your regression you get a linear predictor for each case based on the sum of the products of predictor values with their coefficients. The way you have coded the problem, the linear predictor values for the indicated number of attempts is:

  1. $\beta_1$
  2. $2\beta_1+\beta_2$
  3. $3\beta_1+\beta_2+\beta_3$
  4. $4\beta_1+\beta_2+\beta_3+\beta_4$
  5. $5\beta_1+\beta_2+\beta_3+\beta_4$

That's not invalid, it's just represents another implicit assumption, in this case about the form of weighting different numbers of attempts.

You might consider treating the number of attempts as an ordinal predictor; answers on this page suggest several ways to proceed, with orthogonal polynomials or spline fits for the number of attempts. However you code the predictors, though, emphasize the overall quality of the probability model, and base any choice of cutoff on a clinically indicated rather than a "statistically signficant" tradeoff between specificity and sensitivity.

For example, in your logistic regression model don't use the statistical significance of the individual regression coefficients to choose the cutoff, but use the predicted probability to help inform the choice of whether to proceed. In practice, as much as clinicians might sometimes claim to want a cutoff, they typically include (sometimes unconsciously) lots of probabilities about various clinical aspects as they make their decisions in individual cases.

Alternate approach: It's not clear to me that your logistic regression model captures all that is going on here. Each trial seems to have 3 potential outcomes: success at unclogging, failure at unclogging, or an undesired result (injuring blood vessel, device breakage). After the first and last of these outcomes there are presumably no more attempts. If there is a simple failure to unclog, then the clinician decides whether to make another attempt.

So you don't know whether unclogging was impossible unless the clinician kept on trying until there was an undesired result. Thus your logistic regression might actually be modeling the probability of a clinician's giving up rather than an inherent property of clogged vessels.

You might instead consider modeling these probabilities of outcomes and the choice of whether to continue following a simple failure to unclog, as functions of the number of attempts. For example, are these probabilities constant, or do they change with the number of attempts? After how many attempts is the probability of unclogging at the next attempt offset by the risk (and costs) of an undesired outcome? That could be more useful than a logistic regression model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.