I have a data-set (15-20k) with two classes. I can train a classifier on both classes, but am only allowed to test/predict on one class. The data-set is not balanced (~1:4).
I want to find out, how much the classifier was able to learn from the data-set and am therefore i am interested in the predicted probabilities of that one class I can test on resp. their "distribution".
The TPR, for example exists, but uses only the predicted labels (not the "probabilities"). Having not well balanced sets and not calibrated classifiers, this does not seem to be optimal.
Is there a good metric available, that takes the predicted "probabilities" (without calibration, we may don't even speak of probabilities...) of only one class (+ true label) and returns a meaning-full score? Or is it possible to calibrate the output of a classifier by using only one class to test on (so that the predictions are more meaning-full)?