13
$\begingroup$

I have a dataset with two classes of elements. I also have two methods which assign (complementary) probabilities to each element in the dataset of belonging to either class.

Given that I work with probabilities (instead of hard 0,1 classification values), I was pointed to scoring rules as a way to asses which method performs better. The two most used rules appear to be:

with Log loss apparently being the standard approach (is it?). I also found scikit-learn's roc_auc_score, an implementation of the:

which appears to do pretty much the same thing.

My question is: is either one of these inherently "better" than the other in some form? I also could just use all three. Is this advisable?

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
14
$\begingroup$

The choice depends on how you plan to use the model. There are many potential strictly proper scoring rules (AUC isn't one). They effectively put different weights on different parts of the probability scale while still all meeting the requirement of having an optimal value at the true probabilities.

I have found the report "Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications," by Andreas Buja, Werner Stuetzle, and Yi Shen, to be very helpful in thinking about this. The authors show that choice of probability cutoff is equivalent to a choice of the relative cost of false-positive and false-negative classifications. They then provide a way to tailor loss functions to meet different choices of relative costs.

So the choice of scoring rule might best take the eventual use of the model into account. For a bit more detail without going into that full 48-page report, see related answers here and here.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ +1 Just note that AUC-ROC (a semi-proper scoring rule) is far from catastrophic at first instance. Finding the true DGP is not necessity of a good prediction/ a Bayes optimal classifier. $\endgroup$
    – usεr11852
    Commented Jun 6, 2020 at 13:18
  • $\begingroup$ @usεr11852 what do you mean by "is far from catastrophic"? Also, what does DGP mean? $\endgroup$
    – Gabriel
    Commented Jun 6, 2020 at 13:31
  • 1
    $\begingroup$ @usεr11852 I had intended to write "strictly proper scoring rule" as a qualifier before "AUC isn't one" and have now edited thusly. I agree that AUC isn't necessarily a terrible choice. Frank Harrell does frequently note on this site that it's not as sensitive as other measures for distinguishing between models. $\endgroup$
    – EdM
    Commented Jun 6, 2020 at 13:39
  • 3
    $\begingroup$ @Gabriel: Apologies for any confusion. 1. I mean that in comparison with scoring rules that might be outright misleading (e.g. Recall where taken on its own is almost nonsensical), AUC-ROC (ie. the Mann-Whitney U-Test), while not as discriminant as Brier score, is by and large informative and reliable. 2."DGP" stands for Data Generating Process. 3. Please note that even if we use only strictly proper scoring rules (e.g. logarithmic and quadratic), we will not necessarily pick the same model as they penalise mistakes differently. $\endgroup$
    – usεr11852
    Commented Jun 6, 2020 at 21:35
  • $\begingroup$ Thank you very much for the clarification @usεr11852! Also, thank you for pointing out that AUC-ROC is equivalent to the Mann-Whitney U-Test, I was not aware of that. $\endgroup$
    – Gabriel
    Commented Jun 6, 2020 at 22:32
5
$\begingroup$

The problem with the log loss is that it gives an arbitrarily high penalty for getting a single example completely wrong with high confidence. There are ways of dealing with this, but it can make the metric very sensitive to individual samples, which may not be desirable.

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ +1, although this behavior could be considered a feature, not a bug. $\endgroup$
    – Stephan Kolassa
    Commented Mar 1 at 13:05
  • $\begingroup$ @StephanKolassa indeed! I tend to view it more as a bug as a model created from a finite dataset seems rarely justified in being completely certain of anything, so we probably should be interpreting an output of 1 or 0 literally. Sometimes fixing the model, e.g. by marginalising over the uncertainty in the parameters and hyper-parameters is too expensive, so it may be better to fix the metric (which could be viewed as putting a prior on the output of the model to moderate it a bit). It is just something to consider in interpretation. $\endgroup$
    – Dikran Marsupial
    Commented Mar 1 at 13:18
  • 1
    $\begingroup$ Hm. I absolutely agree with "a model created from a finite dataset seems rarely justified in being completely certain of anything", so I would say that if our metric penalizes this spurious certainty heavily, then that is a Good Thing. Which is exactly why I consider this behavior of the log score a feature, not a bug, because it explicitly incentivizes the kind of humility correction you note. $\endgroup$
    – Stephan Kolassa
    Commented Mar 1 at 13:21
  • 1
    $\begingroup$ That is another good point. On the one hand, that could motivate seeing scores (as well as accuracy etc.) as "fuzzy" in themselves. On the other hand, that seems related to "classes" often being the kind of discretization we often warn about here at CV. I understand that psychologists need some kind of definition of Major Depressive Disorder, but looking at the DSM-V or ICD-10 criteria, the MDD diagnosis seems like it discretizes an inherent continuum. $\endgroup$
    – Stephan Kolassa
    Commented Mar 1 at 15:10
  • 1
    $\begingroup$ Yes, a lot of practice is forced by what is implemented in toolboxes (unless you are happy writing your own implementations), and the problem is shoehorned into the form that the toolbox supports. There is a lot to be gained by making models that incorporate more of what you know about the problem instead. $\endgroup$
    – Dikran Marsupial
    Commented Mar 1 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.