0
$\begingroup$

I am currently doing a logistic regression analysis and wanted to cross validate the results. Now that I have the probabillities calculated for each fold, I made a ROC Curve for each one as well. But how do I choose Cut-Off-Values? I want to sum up the classification of all folds and then calculate the accuracy based of the values of all folds. Does it have an impact, when I choose a different Cut-Off Value for each fold, which I chose from each indepent ROC curve? If so, how should I do it?

Thanks in advance :)

Improve this question
$\endgroup$
4
  • $\begingroup$ You want to classify data into two classes, say 0 and 1, and you have a logistic regression model that gives you, for each observation, the probability that this observation should be classified as 1. So you should assign the label 1 whenever this probability is greater than 1/2. What do you need a Cut-Off value for? Maybe I don't understand correctly what you refer to by "Cut-Off value". $\endgroup$
    – frank
    Commented Aug 5, 2022 at 4:57
  • $\begingroup$ @frank A major point behind ROC curves is that you can set the cutoff wherever you want, and the classifications will be different for different cutoff values, resulting in different balances of sensitivity and specificity. In the extreme, we can achieve perfect sensitivity or specificity by sacrificing the other. $\endgroup$
    – Dave
    Commented Aug 5, 2022 at 5:01
  • $\begingroup$ @Dave But the OP is interested in accuracy. $\endgroup$
    – frank
    Commented Aug 5, 2022 at 5:04
  • $\begingroup$ Frank is right. Yes I also want to use the AUC, as well as the accuracy of the the model in total, in order to compare it with another method. And the accuracy is dependent of the cut-off-value. Therefore, because specifity and sensintivity differs with the folds, should I pick a value based on a qualitative decision for every fold? $\endgroup$
    – MauM99
    Commented Aug 5, 2022 at 8:22

2 Answers 2

Reset to default
2
$\begingroup$

  • substantial variation in the cutoffs determined via cross validation tell you that your cutoff estimation algorithm does not yield stable results.

  • However, that may just be because of small numbers of test cases in each fold. Do make sure you find out what the root cause behind the instability is, though.

  • Keep in mind: When you determine cutoff via CV, the "test" cases become part of the final model's training procedure. To estimate generalization performance of the final model, you thus need test data that are independent of the cutoff determination as well as of the model parameter determination.
    If you observe a drop in performance with that, your cutoff determination overfits.

  • Side note: accuracy is rarely the figure of metric that matters from the application perspective.

Improve this answer
$\endgroup$
3
  • $\begingroup$ 1. What do you mean by "variation in the cutoffs determined via cross validation"? I would pick the values based on sensitivity and specifity. Do you mean if I get cut-offs that strongly differ despite equal sensitivity and specifity? I got a set of 1000 entries, if that helps. 2. Should I split each fold into trainining, validating (for the AUC) and testing then? 3. What indicators next to AUC would you recommend? Thank you for your answer :) $\endgroup$
    – MauM99
    Commented Aug 5, 2022 at 13:10
  • $\begingroup$ @MauM99: yes: you set yourself some rule how to determine the cutoff, but for different folds (and for the combined predictions of all folds) the cutoff predicted probabiliy differs. $\endgroup$
    – cbeleites
    Commented Aug 5, 2022 at 13:14
  • $\begingroup$ Ah, I see. If you have the time could you also answer the other questions I added to the comment? :) $\endgroup$
    – MauM99
    Commented Aug 5, 2022 at 13:28
1
$\begingroup$

Choosing cutoffs in general, and in particular based on indexes derived from retrospective sampling (ROC curve, sens, spec), is a process that is completely at odds with decision making, which is a fully prospective process (predict the future based on current and past data). Optimum decisions have absolutely nothing to do with the metrics you are using. For details see https://www.fharrell.com/post/mlconfusion/

If I wanted to estimate the probability that a left-handed baseball play gets a hit I'd compute P(hit | batter left) and would have no interested in the sens/spec idea of P(better left handed | gets a hit).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.