# How to estimate the optimal cutpoint for a binary outcome in python?

I have a dataset of diabetic patients which has been used to train an xgboost model in several outcomes such as stroke, amputation, and more. Originally we used the continuous numeric variables as-is, but we found ambiguity in the results since for example age was giving us results where the older you get the higher the risk to have a stroke.

But, for us as physicians we need a narrower range, so we divide those variables in bins. And indeed this gave us more insight. Nonetheless, we are seeing that some contiguous intervals appear in our results pretty close.

Continuing from the example above, bin(64-78) and bin(79-88), appear one after the other and no other bin from the age variable appears. So we think that the best approach, in this case, is to find the best optimal cutpoint at which the age starts to become a risk factor for stroke.

Then I came across this document (https://www.mayo.edu/research/documents/biostat-79pdf/doc-10027230) which explains in SAS how to find those cutpoints. I am not experienced enough to program this myself, so I want to know how could I achieve to find these cutpoints in python?

Do please restrict to that language, I have already seen R, SAS, even SPSS examples but none in python. There must be some way to do this in Python.

• Don't bin your continuous data. Feed them into your algorithm as-is; potentially transform them using (e.g.) restricted cubic splines (see, e.g., Frank Harrell's Regression Modeling Strategies) to capture any nonlinearity. – Stephan Kolassa Sep 23 '20 at 15:24
• There are a number of methods with the common name optimal binning aka supervised binning. Read about it. Though binning of a continuous predictor is often not recommended, sometimes binning is the goal, and sometimes a subsequent analysis demands it be done. – ttnphns Sep 23 '20 at 15:38
• The problem is that if I use the variables as-is it takes me to conclusions such as the older a person is the more likely they will have a stroke, that is already studied, I want to know from which age the risk starts, and so on for all the other variables. Binning IS NOT the problem I want the BEST OPTIMAL CUT-OFF POINT (max sensitivity & specificity) for an outcome, I know how to do it in SPSS i just don't know how to do it in python, I would need a ROC curve and that would give me the thresholds and from there I can select MANUALLY the value, which is why I don't want this – alexzaizar09 Sep 24 '20 at 3:20
• "I want to know from which age the risk starts" - this does not make sense. There is no one point in time at which risk suddenly goes from zero to some nonzero value. It's a continuum. Binning does not make sense here. – Stephan Kolassa Sep 30 '20 at 14:24
• Regarding sensitivity and specificity: don't use them. Every criticism leveled at accuracy at Why is accuracy not the best measure for assessing classification models? applies equally to sensitivity and specificity. – Stephan Kolassa Sep 30 '20 at 14:26

If the relationship between continuous age and stroke is already known and well estimated, then you are not adding any scientific knowledge with any kind of binning. In particular, if there were points where the risk of stroke suddenly changes, a sufficiently flexible continuous model (such as splines) would have captured them.

I am no expert on stroke specifically, but here are two illustrations that a quick search gave me:

This is of course not a replacement for a proper analysis, and some of the smoothness comes from factors which can be controlled for, but I do not see anything that looks like an appropriate cutoff in these ranges: concluding that people <64 years have no risk of stroke, for example, would be clearly false here.

On the other hand, there may be legitimate cases when binning is needed, such as when subsequent analysis/decision making allows only two groups. However, from the comments, this does not seem to be the case:

I want the BEST OPTIMAL CUT-OFF POINT (max sensitivity & specificity) for an outcome

Both sensitivity and specificity cannot be maximized at the same point unless there is something extremely weird going on with the thresholding. You will have to make a tradeoff between how bad is it to miss a case (sensitivity) and how bad is a false positive (specificity). Packages such as SPSS may have their own default weights to solve this; relying on that to design any real-life application would be grossly irresponsible and potentially endangering lives, in my opinion.

Basically, if you can assign a cost value to each of true/false positives/negatives based on the purpose of having these groups, then we can propose a way to minimize that cost. Otherwise, any randomly chosen cutoff can be argued to be "optimal" for some cost.

On top of that, there is a range of other problems with using sensitivity / specificity in practice, e.g. https://www.academicradiology.org/article/S1076-6332(03)80087-9/abstract and elsewhere.

In short - any single cutpoint is probably not optimal for what you're trying to achieve, hence we can't recommend a procedure to find it at this stage.

• What we have found is that some of those publications MAY have a correlation between the number of patients in each age bin with the amount of the outcome studied. That is exactly what we want to exclude here, we have already stratified evenly across all bins and data does not follow those curves it has peaks and valleys. – alexzaizar09 Sep 26 '20 at 0:15
• @alexzaizar09 not sure I follow, is different $n$ in each bin a problem?.. Secondly, variability of the outcome inside each bin (if that's what you're referring to) is again a symptom that bins are insufficient to adjust for the age effect, and continuous data should be used. You might want to edit the original post to include any other relevant info. – juod Sep 26 '20 at 0:47
• @joud I could not come up with something to re-write, I have several continuous variables binned at different n, according to their range of values or reported literature, the outcome in ALL cases is binary. I want to convert to binary those continuous variables instead of bins. is that clearer? – alexzaizar09 Sep 26 '20 at 16:19
• @alexzaizar09 converting a continuous predictor variable into binary is just converting it into 2 bins. All the same principles (and dangers) apply. – EdM Sep 28 '20 at 15:20