# Binary classification: single label probability based metric/calibration

Situation

I have a data-set (15-20k) with two classes. I can train a classifier on both classes, but am only allowed to test/predict on one class. The data-set is not balanced (~1:4).

Goal

I want to find out, how much the classifier was able to learn from the data-set and am therefore i am interested in the predicted probabilities of that one class I can test on resp. their "distribution".

Problem

The TPR, for example exists, but uses only the predicted labels (not the "probabilities"). Having not well balanced sets and not calibrated classifiers, this does not seem to be optimal.

Question

Is there a good metric available, that takes the predicted "probabilities" (without calibration, we may don't even speak of probabilities...) of only one class (+ true label) and returns a meaning-full score? Or is it possible to calibrate the output of a classifier by using only one class to test on (so that the predictions are more meaning-full)?

• look at the gain curve: %of population vs %of events captured when rank ordered decending. perhaps, this might be what you are looking for – muni Dec 6 '16 at 12:32

I recommend you look into cost curves. These (shown on the right of the figure below) display the normalized expected cost (i.e., error) at different probability costs (i.e., class probability or cost function). This will not give a single score necessarily but will show the range of performance. Drummond, C., & Holte, R. C. (2006). Cost curves: An improved method for visualizing classifier performance. Machine Learning, 65(1), 95–130.

• That's looks actually nice and I was not aware of this metric, but it does not answer my question. As far as I understand things, you still need both classes in the test sample and cannot use the predicted probabilities of only one class... – Mayou36 Dec 2 '16 at 9:52

1) Almost all classifiers will also return a score for each prediction - some measure of "certainty" on that prediction. I will use examples from sklearn but most of what I mention here is available in other languages/frameworks

In sklearn, the method predict_proba of any classifier ( see for example RandomForests) return a measure of certainty on the classification. Although this is called a probability (because it is a number between 0 and 1) it is not really a probability. So you can interpret that number as a real probability you have to calibrate it - there are two calibration algorithms implemented in sklearn Platt's and isotonic.

At the end of the calibration, you want to be able to make statements such as "in 80% of the cases in which the classifier predicts that the data is positive with probability 80% , the classifier is correct". That is, when the classifier makes a prediction with x% probability, it is right x% of the cases.

2) You can and must do all the learning and calibration on the training set, where you have the 2 classes. But you will not be able to verify that the learning or the calibration is "good" on your test set, if it contains only one class. There may be some metrics that allow one to compare a probabilistic prediction with a binary outcome - but my feeling is that given that the test set has only one class, this measure will not be very informative.

EDIT

3) The wikipedia Scoring rules is about different metrics that score a probabilistic prediction with the outcome.

• Thanks for your answer, but actually I knew that already. My question starts where your answer ends: what kind of measure can we use for that? And the calibration does not work this way: you do CV which means you will make predictions on both class labels... – Mayou36 Dec 6 '16 at 13:48
• Yes, for calibration you do CV but on the training set, not the test set – Jacques Wainer Dec 6 '16 at 16:02
• But my restriction is, that I can only predict one class and not the other one... the simple reason is, that one class has weights and prediction then may induces a bias. – Mayou36 Dec 6 '16 at 20:09