8
$\begingroup$

I have a dataset with two classes of elements. I also have two methods which assign (complementary) probabilities to each element in the dataset of belonging to either class.

Given that I work with probabilities (instead of hard 0,1 classification values), I was pointed to scoring rules as a way to asses which method performs better. The two most used rules appear to be:

with Log loss apparently being the standard approach (is it?). I also found scikit-learn's roc_auc_score, an implementation of the:

which appears to do pretty much the same thing.

My question is: is either one of these inherently "better" than the other in some form? I also could just use all three. Is this advisable?

$\endgroup$

2 Answers 2

10
$\begingroup$

The choice depends on how you plan to use the model. There are many potential strictly proper scoring rules (AUC isn't one). They effectively put different weights on different parts of the probability scale while still all meeting the requirement of having an optimal value at the true probabilities.

I have found the report "Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications," by Andreas Buja, Werner Stuetzle, and Yi Shen, to be very helpful in thinking about this. The authors show that choice of probability cutoff is equivalent to a choice of the relative cost of false-positive and false-negative classifications. They then provide a way to tailor loss functions to meet different choices of relative costs.

So the choice of scoring rule might best take the eventual use of the model into account. For a bit more detail without going into that full 48-page report, see related answers here and here.

$\endgroup$
5
  • $\begingroup$ +1 Just note that AUC-ROC (a semi-proper scoring rule) is far from catastrophic at first instance. Finding the true DGP is not necessity of a good prediction/ a Bayes optimal classifier. $\endgroup$
    – usεr11852
    Jun 6, 2020 at 13:18
  • $\begingroup$ @usεr11852 what do you mean by "is far from catastrophic"? Also, what does DGP mean? $\endgroup$
    – Gabriel
    Jun 6, 2020 at 13:31
  • $\begingroup$ @usεr11852 I had intended to write "strictly proper scoring rule" as a qualifier before "AUC isn't one" and have now edited thusly. I agree that AUC isn't necessarily a terrible choice. Frank Harrell does frequently note on this site that it's not as sensitive as other measures for distinguishing between models. $\endgroup$
    – EdM
    Jun 6, 2020 at 13:39
  • 3
    $\begingroup$ @Gabriel: Apologies for any confusion. 1. I mean that in comparison with scoring rules that might be outright misleading (e.g. Recall where taken on its own is almost nonsensical), AUC-ROC (ie. the Mann-Whitney U-Test), while not as discriminant as Brier score, is by and large informative and reliable. 2."DGP" stands for Data Generating Process. 3. Please note that even if we use only strictly proper scoring rules (e.g. logarithmic and quadratic), we will not necessarily pick the same model as they penalise mistakes differently. $\endgroup$
    – usεr11852
    Jun 6, 2020 at 21:35
  • $\begingroup$ Thank you very much for the clarification @usεr11852! Also, thank you for pointing out that AUC-ROC is equivalent to the Mann-Whitney U-Test, I was not aware of that. $\endgroup$
    – Gabriel
    Jun 6, 2020 at 22:32
4
$\begingroup$

The problem with the log loss is that it gives an arbitrarily high penalty for getting a single example completely wrong with high confidence. There are ways of dealing with this, but it can make the metric very sensitive to individual samples, which may not be desirable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.