Performance metric for continuous binary classification method

Question

I have and imbalanced data set with two classes of data: $A$ and $B$. I apply a method that assigns a continuous probability to each element of belonging to class $A$: $P_{A}$ , where $P_B=1-P_A$.

I need a way to assess its performance, but all the metrics I've found assume that the result of your classification method is either 1 or 0 ($A$ or $B$):

• 24 Evaluation Metrics for Binary Classification (And When to Use Them)
• Evaluating Classification Models
• The ultimate guide to binary classification metrics
• Metrics and scoring: quantifying the quality of predictions

I could "transform" my results to this format by splitting $P_A$ at $P_A=0.5$ and assuming larger values are 1 (element classified as $A$), and smaller values are 0 (element classified as $B$), but this feels like I'm throwing information away.

Is there a metric that makes use of the fact that I have a continuous range of probabilities and not just 1 or 0?

Answer 1

You are looking for scoring-rules, which do precisely what you want: they assess the quality probabilistic predictions. Specifically, you want proper scoring rules, which are scoring rules that are optimized on the "correct" probabilistic predictions. Here is Wikipedia.

The simplest one you could use would be the logarithmic score, which is $\log P_A$ if the class turns out to be $A$, and $\log P_B$ if it is $B$. Alternatively, you can look at the Brier score. Or others given at the Wikipedia site.

There are a number of papers by Tilmann Gneiting and colleagues that will give you a lot of theoretical background. For instance, Ehm et al. (2016, JRSS B) unifies the different possible scoring rules into one framework.

In any case: no, don't discretize your predictions. There are a lot of misleading suggestions floating around out there, unfortunately. And unbalanced data are not a problem if you know what you are doing.