Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset

I am currently doing a logistic regression analysis and wanted to cross validate the results. Now that I have the probabillities calculated for each fold, I made a ROC Curve for each one as well. But how do I choose Cut-Off-Values? I want to sum up the classification of all folds and then calculate the accuracy based of the values of all folds. Does it have an impact, when I choose a different Cut-Off Value for each fold, which I chose from each indepent ROC curve? If so, how should I do it?

• You want to classify data into two classes, say 0 and 1, and you have a logistic regression model that gives you, for each observation, the probability that this observation should be classified as 1. So you should assign the label 1 whenever this probability is greater than 1/2. What do you need a Cut-Off value for? Maybe I don't understand correctly what you refer to by "Cut-Off value". Aug 5, 2022 at 4:57
• @frank A major point behind ROC curves is that you can set the cutoff wherever you want, and the classifications will be different for different cutoff values, resulting in different balances of sensitivity and specificity. In the extreme, we can achieve perfect sensitivity or specificity by sacrificing the other.
– Dave
Aug 5, 2022 at 5:01
• @Dave But the OP is interested in accuracy. Aug 5, 2022 at 5:04
• Frank is right. Yes I also want to use the AUC, as well as the accuracy of the the model in total, in order to compare it with another method. And the accuracy is dependent of the cut-off-value. Therefore, because specifity and sensintivity differs with the folds, should I pick a value based on a qualitative decision for every fold? Aug 5, 2022 at 8:22

• substantial variation in the cutoffs determined via cross validation tell you that your cutoff estimation algorithm does not yield stable results.

• However, that may just be because of small numbers of test cases in each fold. Do make sure you find out what the root cause behind the instability is, though.

• Keep in mind: When you determine cutoff via CV, the "test" cases become part of the final model's training procedure. To estimate generalization performance of the final model, you thus need test data that are independent of the cutoff determination as well as of the model parameter determination.
If you observe a drop in performance with that, your cutoff determination overfits.

• Side note: accuracy is rarely the figure of metric that matters from the application perspective.

• 1. What do you mean by "variation in the cutoffs determined via cross validation"? I would pick the values based on sensitivity and specifity. Do you mean if I get cut-offs that strongly differ despite equal sensitivity and specifity? I got a set of 1000 entries, if that helps. 2. Should I split each fold into trainining, validating (for the AUC) and testing then? 3. What indicators next to AUC would you recommend? Thank you for your answer :) Aug 5, 2022 at 13:10
• @MauM99: yes: you set yourself some rule how to determine the cutoff, but for different folds (and for the combined predictions of all folds) the cutoff predicted probabiliy differs. Aug 5, 2022 at 13:14
• Ah, I see. If you have the time could you also answer the other questions I added to the comment? :) Aug 5, 2022 at 13:28

Choosing cutoffs in general, and in particular based on indexes derived from retrospective sampling (ROC curve, sens, spec), is a process that is completely at odds with decision making, which is a fully prospective process (predict the future based on current and past data). Optimum decisions have absolutely nothing to do with the metrics you are using. For details see https://www.fharrell.com/post/mlconfusion/

If I wanted to estimate the probability that a left-handed baseball play gets a hit I'd compute P(hit | batter left) and would have no interested in the sens/spec idea of P(better left handed | gets a hit).