3
$\begingroup$

I am working with a dataset of diseased vs non-diseased patients. I'm looking for a cut-off value for determining the status of a new patient. This is intended for lab technicians and MDs to allow for quick diagnosis. It's got to be practical, so please spare me from discussing whether it's the right thing to do—I already know it's not a favorable approach.

To be as thorough as possible I want to use multiple methods to determine the cut-off. My hope is that these methods will yield similar values. So far I've used logistic regression (ROC) and mixture modeling. So my question: what other techniques can be used to determine the cut-off?

1/10/2014 EDIT: To be clear - I actually have 3 variables to be classified in the dataset, however only one of them is practically useful. The other two are highly correlated with one another, and not really correlated with the outcome. If I have to use more than one, I will use a ratio, so no need for multivariate classifiers. The goal is to be simple - just one cut-off value.

$\endgroup$
  • 1
    $\begingroup$ To be clear, you are classifying based on a single lab value? or multiple variables? $\endgroup$ – Ellis Valentiner Jan 9 '14 at 20:55
  • 2
    $\begingroup$ Classification tree(s) and discriminant analysis are two more techniques that might be appropriate. $\endgroup$ – Jean V. Adams Jan 9 '14 at 22:56
  • $\begingroup$ You likely need to provide more information to get a reasonable answer. Also worth noting that any classification method will need to be coupled with some utility function (FP/FN) to determine a cut-off. $\endgroup$ – charles Jan 9 '14 at 23:14
  • 1
    $\begingroup$ @MichalJ.Figurski - You can perform discriminant analysis on one independent variable (x). The dependent variable (y) is categorical, in your case it's patient status (diseased or non-diseased). $\endgroup$ – Jean V. Adams Jan 13 '14 at 13:16
  • 1
    $\begingroup$ @JeanV.Adams - I somehow did not realize LDA can be used this way. It works, so now I have 4 methods to choose from. If you re-post your comments as an answer, I'll accept it. Thanks. $\endgroup$ – Michal J. Figurski Jan 13 '14 at 19:43
2
$\begingroup$

Classification tree(s) and discriminant analysis are two more techniques that might be appropriate.

You can perform discriminant analysis on one independent variable (x). The dependent variable (y) is categorical, in your case it's patient status (diseased or non-diseased).

$\endgroup$
  • $\begingroup$ Is it some jealousy going on here? Why would anyone vote down a valid answer? Moreover, an answer that was actually related to the question? Curious... $\endgroup$ – Michal J. Figurski Jan 14 '14 at 19:21
3
$\begingroup$

You don't need modelling (logistic regression or mixture models) to establish a cut-off. Just compute a ROC curve of your data and it's going to give you cut-offs. To select only one cut-off specifically, use the methods described by Perkins & Schisterman or simply the Youden Index.


  1. Neil J. Perkins, Enrique F. Schisterman (2006) “The Inconsistency of "Optimal" Cutpoints Obtained using Two Criteria based on the Receiver Operating Characteristic Curve”. American Journal of Epidemiology 163(7), 670–675. DOI: 10.1093/aje/kwj063.
  2. W. J. Youden (1950) “Index for rating diagnostic tests”. Cancer, 3, 32–35. DOI: 10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3.
$\endgroup$
  • $\begingroup$ I think logistic regression is actually a part of ROC, just for one variable. I already did that - now I'm looking for other ways to determine cutoff. $\endgroup$ – Michal J. Figurski Jan 10 '14 at 16:28
  • 1
    $\begingroup$ @MichalJ.Figurski No it is not. You can make a ROC curve out of a logistic regression model, but that's not a requirement. You can do a ROC curve or of any other model, or you can simply make a ROC curve of your data. $\endgroup$ – Calimo Jan 10 '14 at 16:38
  • 1
    $\begingroup$ ROC...and utility/cost function. How does an ROC curve give you a cut-off without a utility/cost function? $\endgroup$ – charles Jan 11 '14 at 0:44
  • 1
    $\begingroup$ Ok it's going to give you several cut-offs, and then you'll have to pick one - I'll update my answer with cost functions to do that. $\endgroup$ – Calimo Jan 11 '14 at 14:46
  • $\begingroup$ Guys, this is pointless - I'm looking for ways to determine cutoff other than ROC. $\endgroup$ – Michal J. Figurski Jan 13 '14 at 19:40
2
$\begingroup$

I did similar classification problem last fall in the context of churn analysis with our CRM data.

I used logistic regression, neural networks and decision trees.

ROC-analysis is one method for evaluation of probabilistic classifications from several models, it is not tied to logistic regression and it uses all cut-offs from range [0,1]. But this was just a start of analysis.

Next method for evaluation of classifications was setting explicit cost-values for false positives and false negatives. Then I just decided to make a cut-off at point which minimizes total costs.

Reason why I wanted to have explicit cost values was that our business users could say to me what is relative importance of different false classifications with respect to each other.

Can your business users say what is relative importance of various cells in the confucion matrix?

$\endgroup$
  • 1
    $\begingroup$ In my case the goal is to find optimal balance of sensitivity and specificity. The group comes from a biomed cohort study, and the cut-off is designed to help in future cohort studies - not for general population. In this case I select the point of max(sens+spec) from ROC. $\endgroup$ – Michal J. Figurski Jan 14 '14 at 16:51
  • 1
    $\begingroup$ @MichalJ.Figurski, this may be good loss-function. I had other information from business users which I then used for decision making. ROC itself cannot tell that, you need something. But ROC can help to decide if the method itself can provide some firepower for the classification problem. $\endgroup$ – Analyst Jan 15 '14 at 6:19
  • 1
    $\begingroup$ @Analyst has given the best answer so far. FN may equal FP, but you have to accept some utility function before choosing cutoff. $\endgroup$ – charles Jan 18 '14 at 4:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.