# Binary classification cross validation ROC score - only consider higher confidence class probabilities

Given a binary classification task, during 10-fold cross-validation I'm able to get the probability that each test set sample is of one class or the other.

When I compute the AUC ROC score during cross validation, the score is quite consistently 0.7 for each of the ten folds when using the straightforward approach of just assigning classes to the test set samples if the probability is > 0.5

I was curious, however, to see if the ROC score could be improved by only assigning a class if the probability assigned is greater than some arbitrary cutoff (like .8), then rescoring.

When I do this, I lose about 85% of the test samples, but the resulting ROC score of the high confidence test set is boosted to 0.87, which makes it useful for downstream analysis.

In this particular case I would:

• Discard test samples with prediction probabilities below the cutoff
• For one class, use regression to make predictions
• For the other class, leave it as it is

(As background information, the data for this classification task was binarized from data where most y values are zero. I had no success using regression, so first I'll use classification to determine which samples are zero, then do regression on the rest. The regression approach works quite well when there aren't a ton of zero values in y)

My questions are:

Is this a valid approach to improving the ROC score? I can't see any reason why not but ML is not my specialty and I might be missing something.

If it is valid, do I have to watch out for any class imbalances in the resulting high confidence test set when computing the ROC score?

• You are presumably building a prediction model for some other purpose than just getting a high AUROC. In view of that purpose, what would you do with instances with a low confidence? Sep 28, 2022 at 11:51
• @StephanKolassa question edited for clarity
– Ryan
Sep 28, 2022 at 11:57
• There is precedent in caring only about the ROC curve for certain ranges of fpr, called "partial auc/roc" in some packages: pubmed.ncbi.nlm.nih.gov/2668680 Sep 28, 2022 at 12:38
• "When I compute the AUC ROC score [...] using the straightforward approach of just assigning classes to the test set samples if the probability is > 0.5". I don't understand this bit: the ROC curve is generated by varying that decision threshold. Sep 28, 2022 at 12:40
• @BenReiniger when using what I call the "straightforward" approach, I pass y_test_true and y_test_predictions, to roc_auc_score and get a AUROC of 0.7. When I consider probabilities, I only take predictions where the confidence is above a threshold, then rescore based on that high confidence subset. Maybe I am not understanding something, but I think the scoring function is unable to consider probabilities because I am only passing binarized data to it. Or at least the decision function used in the ROC curve is different than the decision function I use to define the high confidence set
– Ryan
Sep 28, 2022 at 13:14