Binary classification cross validation ROC score - only consider higher confidence class probabilities

Question

Given a binary classification task, during 10-fold cross-validation I'm able to get the probability that each test set sample is of one class or the other.

When I compute the AUC ROC score during cross validation, the score is quite consistently 0.7 for each of the ten folds when using the straightforward approach of just assigning classes to the test set samples if the probability is > 0.5

I was curious, however, to see if the ROC score could be improved by only assigning a class if the probability assigned is greater than some arbitrary cutoff (like .8), then rescoring.

When I do this, I lose about 85% of the test samples, but the resulting ROC score of the high confidence test set is boosted to 0.87, which makes it useful for downstream analysis.

In this particular case I would:

• Discard test samples with prediction probabilities below the cutoff
• For one class, use regression to make predictions
• For the other class, leave it as it is

(As background information, the data for this classification task was binarized from data where most y values are zero. I had no success using regression, so first I'll use classification to determine which samples are zero, then do regression on the rest. The regression approach works quite well when there aren't a ton of zero values in y)

My questions are:

Is this a valid approach to improving the ROC score? I can't see any reason why not but ML is not my specialty and I might be missing something.

If it is valid, do I have to watch out for any class imbalances in the resulting high confidence test set when computing the ROC score?

Answer 1

When I do this, I lose about 85% of the test samples, but the resulting ROC score of the high confidence test set is boosted to 0.87, which makes it useful for downstream analysis.

It sounds to me like you are simply changing the decision threshold in a way that I discuss in earlier answerss of mine here and here: If you are confident in your predictions, take certain actions, and if you are not so confident (predictions farther away from 0 and 1), take yet other actions, e.g., collecting more data or running more clinical tests.

That is a natural and very useful way of dealing with uncertainty. I would just not cast it as "improved AUROC", but in terms of separating the statistical prediction aspect and the subsequent decision/action step, as I advocated in the two threads linked above.

Answer 2

The ROC curve and corresponding AUC relate to assessing performance across a spectrum of thresholds, not performance at one particular threshold.

Consequently, your plan seems to be equivalent to excluding the low-confidence predictions and then evaluating performance on only the examples where you are confident in your answer.

“Professor, don’t grade questions two or three, since I didn’t know how to do them,” is likely to raise an exam score, yes, but it does not indicate improved knowledge of the subject.