4
$\begingroup$

Given a binary classification task, during 10-fold cross-validation I'm able to get the probability that each test set sample is of one class or the other.

When I compute the AUC ROC score during cross validation, the score is quite consistently 0.7 for each of the ten folds when using the straightforward approach of just assigning classes to the test set samples if the probability is > 0.5

I was curious, however, to see if the ROC score could be improved by only assigning a class if the probability assigned is greater than some arbitrary cutoff (like .8), then rescoring.

When I do this, I lose about 85% of the test samples, but the resulting ROC score of the high confidence test set is boosted to 0.87, which makes it useful for downstream analysis.

In this particular case I would:

  • Discard test samples with prediction probabilities below the cutoff
  • For one class, use regression to make predictions
  • For the other class, leave it as it is

(As background information, the data for this classification task was binarized from data where most y values are zero. I had no success using regression, so first I'll use classification to determine which samples are zero, then do regression on the rest. The regression approach works quite well when there aren't a ton of zero values in y)

My questions are:

Is this a valid approach to improving the ROC score? I can't see any reason why not but ML is not my specialty and I might be missing something.

If it is valid, do I have to watch out for any class imbalances in the resulting high confidence test set when computing the ROC score?

$\endgroup$
7
  • 2
    $\begingroup$ You are presumably building a prediction model for some other purpose than just getting a high AUROC. In view of that purpose, what would you do with instances with a low confidence? $\endgroup$ Sep 28, 2022 at 11:51
  • $\begingroup$ @StephanKolassa question edited for clarity $\endgroup$
    – Ryan
    Sep 28, 2022 at 11:57
  • 1
    $\begingroup$ There is precedent in caring only about the ROC curve for certain ranges of fpr, called "partial auc/roc" in some packages: pubmed.ncbi.nlm.nih.gov/2668680 $\endgroup$ Sep 28, 2022 at 12:38
  • 3
    $\begingroup$ "When I compute the AUC ROC score [...] using the straightforward approach of just assigning classes to the test set samples if the probability is > 0.5". I don't understand this bit: the ROC curve is generated by varying that decision threshold. $\endgroup$ Sep 28, 2022 at 12:40
  • $\begingroup$ @BenReiniger when using what I call the "straightforward" approach, I pass y_test_true and y_test_predictions, to roc_auc_score and get a AUROC of 0.7. When I consider probabilities, I only take predictions where the confidence is above a threshold, then rescore based on that high confidence subset. Maybe I am not understanding something, but I think the scoring function is unable to consider probabilities because I am only passing binarized data to it. Or at least the decision function used in the ROC curve is different than the decision function I use to define the high confidence set $\endgroup$
    – Ryan
    Sep 28, 2022 at 13:14

2 Answers 2

5
$\begingroup$

When I do this, I lose about 85% of the test samples, but the resulting ROC score of the high confidence test set is boosted to 0.87, which makes it useful for downstream analysis.

It sounds to me like you are simply changing the decision threshold in a way that I discuss in earlier answerss of mine here and here: If you are confident in your predictions, take certain actions, and if you are not so confident (predictions farther away from 0 and 1), take yet other actions, e.g., collecting more data or running more clinical tests.

That is a natural and very useful way of dealing with uncertainty. I would just not cast it as "improved AUROC", but in terms of separating the statistical prediction aspect and the subsequent decision/action step, as I advocated in the two threads linked above.

$\endgroup$
5
$\begingroup$

The ROC curve and corresponding AUC relate to assessing performance across a spectrum of thresholds, not performance at one particular threshold.

Consequently, your plan seems to be equivalent to excluding the low-confidence predictions and then evaluating performance on only the examples where you are confident in your answer.

“Professor, don’t grade questions two or three, since I didn’t know how to do them,” is likely to raise an exam score, yes, but it does not indicate improved knowledge of the subject.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.