I have and imbalanced data set with two classes of data: $A$ and $B$. I apply a method that assigns a continuous probability to each element of belonging to class $A$: $P_{A}$ , where $P_B=1-P_A$.

I need a way to assess its performance, but all the metrics I've found assume that the result of your classification method is either 1 or 0 ($A$ or $B$):

I could "transform" my results to this format by splitting $P_A$ at $P_A=0.5$ and assuming larger values are 1 (element classified as $A$), and smaller values are 0 (element classified as $B$), but this feels like I'm throwing information away.

Is there a metric that makes use of the fact that I have a continuous range of probabilities and not just 1 or 0?


You are looking for , which do precisely what you want: they assess the quality probabilistic predictions. Specifically, you want proper scoring rules, which are scoring rules that are optimized on the "correct" probabilistic predictions. Here is Wikipedia.

The simplest one you could use would be the logarithmic score, which is $\log P_A$ if the class turns out to be $A$, and $\log P_B$ if it is $B$. Alternatively, you can look at the Brier score. Or others given at the Wikipedia site.

There are a number of papers by Tilmann Gneiting and colleagues that will give you a lot of theoretical background. For instance, Ehm et al. (2016, JRSS B) unifies the different possible scoring rules into one framework.

In any case: no, don't discretize your predictions. There are a lot of misleading suggestions floating around out there, unfortunately. And unbalanced data are not a problem if you know what you are doing.

| cite | improve this answer | |
  • $\begingroup$ Thank you for the detailed answer Stephan. One thing is not clear to me: are these scores comparable between datasets? Say I apply the loharithmic score to dataset 1 (D1), obtain the $\log P_A$ or $\log P_B$ value for each element, then I sum all these values (?) to obtain a final score. Now I apply the same method to a completely new dataset D2, which can have more or less elements and a different balance. Are these two final scores comparable? Can I say: "the method worked better on D2 over D1 because the score is larger", or is some normalization required? $\endgroup$ – Gabriel Feb 17 at 17:43
  • 1
    $\begingroup$ No, scores are typically not comparable between datasets. They encode how well we can predict, and predictability varies hugely between datasets. This is the exact same effect as what we see when we notice we can't compare MSEs or other accuracy measures between datasets: it's comparing apples and oranges. In both cases, you can only compare different predictions within a single dataset. $\endgroup$ – Stephan Kolassa Feb 18 at 10:06
  • $\begingroup$ That's what I feared. I think I'll investigate a bit and eventually open another question regarding the possible existence of "normalized" scoring rules. Thank you very much again Stephan! $\endgroup$ – Gabriel Feb 18 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.