3
$\begingroup$

Let’s restrict our consideration to binary outcomes.

I have a friend who is terrible at predicting the future, always predicting the opposite of what winds up happening. For instance, my friend predicted Cincinnati to win the Super Bowl, but they lost. My friend predicted that Croatia would beat France for the 2018 World cup, but France won. Et cetera...

That is, my friend's predictions give me excellent predictive accuracy if I go with the opposite of what he predicted. “My friend always gets it wrong, so bet on Los Angeles to beat Cincinnati,” I should have thought a few weeks ago. “Bet on France to beat Croatia,” I should have thought four years ago.

What is wrong with applying this logic to a classification or probability machine learning model that consistently predicts the opposite, measured by $AUC\approx 0?$ I realize that the coefficient estimates should not be trusted, but if we care about the prediction above all else, the danger is not so apparent to me.

That is, if I get $AUC = 0.2$ from the probability predictions $p_i$ of a model, why not take $1-p_i$ as my final predicted probability?

EDIT

$AUC<0.5$, McFadden’s $R^2<0$…whatever metric convinces you that you’re consistently predicting the opposite of what you’re seeing (though I have only ever seen this suggested when the $AUC$ is less than $0.5$, which I’m pretty sure is how some popular software packages approach ROC and AUC calculations).

$\endgroup$
4
  • 1
    $\begingroup$ For binary outcomes... maybe... hard to apply for multi-class. $\endgroup$ Feb 24, 2022 at 13:35
  • $\begingroup$ @user2974951 Yes, it gets complicated for multiple classes. I’ve edited to clarify that I want to consider the binary case. $\endgroup$
    – Dave
    Feb 24, 2022 at 13:43
  • 1
    $\begingroup$ Can I get an easy upvote-and-accept by simply answering "of course, go ahead, do that and have a nice day"? Seriously, your friend is a wonderful predictor, feed them into a model and buy them a cup of coffee. Alternatively, if you need to consider your friend a model and not a predictor, then what they need is just some post-processing, AKA classifier calibration. $\endgroup$ Feb 24, 2022 at 14:01
  • $\begingroup$ @StephanKolassa Not quite, but if you tie this back to calibration, that would make for an interesting answer! (It seems like that would be something like Platt scaling.) $\endgroup$
    – Dave
    Feb 24, 2022 at 14:04

1 Answer 1

0
$\begingroup$

Your question: "That is, if I get $AUC=0.2$ from the probability predictions $p_i$ of a model, why not take $1-p_i$ as my final predicted probability?"

If you did that, then, indeed, your AUC would improve. But your prediction probabilities could be bad. Imagine, all your original prediction probabilities are very high, e.g. $p_i > 1-10^{-6}, i=1,\ldots, N$, but still your $AUC$ is only 0.2 (this is of course possible). Then, when using $1-p_i$, you would predict that nothing is ever happening.

The problem here is that $AUC$ doesn't care about the values of your $p_i$, only about their order.

$\endgroup$
2
  • $\begingroup$ This seems more like a criticism of AUC in general than of taking the opposite probability value. $\endgroup$
    – Dave
    Feb 25, 2022 at 12:29
  • $\begingroup$ @Dave You asked why not take $1-p_i$ when the AUC is 0.2, and I pointed out why not. $\endgroup$
    – frank
    Feb 25, 2022 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.