Binary classification: single label probability based metric/calibration Situation
I have a data-set (15-20k) with two classes. I can train a classifier on both classes, but am only allowed to test/predict on one class. The data-set is not balanced (~1:4).
Goal
I want to find out, how much the classifier was able to learn from the data-set and am therefore i am interested in the predicted probabilities of that one class I can test on resp. their "distribution".
Problem
The TPR, for example exists, but uses only the predicted labels (not the "probabilities"). Having not well balanced sets and not calibrated classifiers, this does not seem to be optimal.
Question
Is there a good metric available, that takes the predicted "probabilities"  (without calibration, we may don't even speak of probabilities...) of only one class (+ true label) and returns a meaning-full score? Or is it possible to calibrate the output of a classifier by using only one class to test on (so that the predictions are more meaning-full)?
 A: I recommend you look into cost curves. These (shown on the right of the figure below) display the normalized expected cost (i.e., error) at different probability costs (i.e., class probability or cost function). This will not give a single score necessarily but will show the range of performance.

Drummond, C., & Holte, R. C. (2006). Cost curves: An improved method for visualizing classifier performance. Machine Learning, 65(1), 95–130.
A: 1) Almost all classifiers will also return a score for each prediction - some measure of "certainty" on that prediction. I will use examples from sklearn but most of what I mention here is available in other languages/frameworks
In sklearn, the method predict_proba of any classifier ( see for example RandomForests) return a measure of certainty on the classification. Although this is called a probability (because it is a number between 0 and 1) it is not really a probability. So you can interpret that number as a real probability you have to calibrate it - there are two calibration algorithms implemented in sklearn Platt's and isotonic. 
At the end of the calibration, you want to be able to make statements such as "in 80% of the cases in which the classifier predicts that the data is positive with probability 80% , the classifier is correct". That is, when the classifier  makes a prediction with x% probability, it is right x% of the cases.
2) You can and must  do all the learning and calibration on the training set, where you have the 2 classes. But you will not be able to verify that the learning or the calibration is "good" on your test set, if it contains only one class. There may be some metrics that allow one to compare a probabilistic prediction with a binary outcome - but my feeling is that given that the test set has only one class, this measure will not be very informative.
EDIT
3) The wikipedia Scoring rules is about different metrics that score a probabilistic prediction with the outcome. 
