Given a binary classification task, during 10-fold cross-validation I'm able to get the probability that each test set sample is of one class or the other.
When I compute the AUC ROC score during cross validation, the score is quite consistently 0.7 for each of the ten folds when using the straightforward approach of just assigning classes to the test set samples if the probability is > 0.5
I was curious, however, to see if the ROC score could be improved by only assigning a class if the probability assigned is greater than some arbitrary cutoff (like .8), then rescoring.
When I do this, I lose about 85% of the test samples, but the resulting ROC score of the high confidence test set is boosted to 0.87, which makes it useful for downstream analysis.
In this particular case I would:
- Discard test samples with prediction probabilities below the cutoff
- For one class, use regression to make predictions
- For the other class, leave it as it is
(As background information, the data for this classification task was binarized from data where most y values are zero. I had no success using regression, so first I'll use classification to determine which samples are zero, then do regression on the rest. The regression approach works quite well when there aren't a ton of zero values in y)
My questions are:
Is this a valid approach to improving the ROC score? I can't see any reason why not but ML is not my specialty and I might be missing something.
If it is valid, do I have to watch out for any class imbalances in the resulting high confidence test set when computing the ROC score?
roc_auc_scoreand get a AUROC of 0.7. When I consider probabilities, I only take predictions where the confidence is above a threshold, then rescore based on that high confidence subset. Maybe I am not understanding something, but I think the scoring function is unable to consider probabilities because I am only passing binarized data to it. Or at least the decision function used in the ROC curve is different than the decision function I use to define the high confidence set $\endgroup$