Performance metric for continuous binary classification method

I have and imbalanced data set with two classes of data: $$A$$ and $$B$$. I apply a method that assigns a continuous probability to each element of belonging to class $$A$$: $$P_{A}$$ , where $$P_B=1-P_A$$.

I need a way to assess its performance, but all the metrics I've found assume that the result of your classification method is either 1 or 0 ($$A$$ or $$B$$):

I could "transform" my results to this format by splitting $$P_A$$ at $$P_A=0.5$$ and assuming larger values are 1 (element classified as $$A$$), and smaller values are 0 (element classified as $$B$$), but this feels like I'm throwing information away.

Is there a metric that makes use of the fact that I have a continuous range of probabilities and not just 1 or 0?

The simplest one you could use would be the logarithmic score, which is $$\log P_A$$ if the class turns out to be $$A$$, and $$\log P_B$$ if it is $$B$$. Alternatively, you can look at the Brier score. Or others given at the Wikipedia site.
• Thank you for the detailed answer Stephan. One thing is not clear to me: are these scores comparable between datasets? Say I apply the loharithmic score to dataset 1 (D1), obtain the $\log P_A$ or $\log P_B$ value for each element, then I sum all these values (?) to obtain a final score. Now I apply the same method to a completely new dataset D2, which can have more or less elements and a different balance. Are these two final scores comparable? Can I say: "the method worked better on D2 over D1 because the score is larger", or is some normalization required? – Gabriel Feb 17 at 17:43