Using a ROC Plot to interpret specific scores I have a binary classifier which outputs a given score to differentiate normal (low score) from abnormal (high score) cases. The score itself however is non-interpretable to others.
I know a ROC plot is typically used to select a threshold to map these continuous scores onto binary decisions. The threshold is selected based on the desired TPR/FPR tradeoff.
My question is: Can this process be done "in reverse", where instead of selecting a binary threshold and them presenting the end user a decision (normal/abnormal), I present the end user the score and say "cases with this score or higher have the following sensitivity (TPR) and specificity (1 - FPR)"
 A: Your model presumably can provide the probability of being in the "abnormal" class as a function of your "non-interpretable" score. A plot of probability versus score would be more useful to users than forcing them to wrap their heads around sensitivity and specificity estimates.
If a user is really forced to make a class assignment based on this single score, then the probability cutoff is based on the relative costs of false-positive and false-negative assignments. If there are other types of information available, the probability based on this score will be most useful to combine with them.
Frank Harrell discusses problems with specificity and sensitivity, and the advantages of direct probability estimates, here.
A: If you only have the scores and no ground truth targets, and you would assign targets according to a chosen threshold for those scores and use those labels as ground truth labels, then you would always get a perfect ROC score equal to one. For the ROC curve to make sense, you need ground truth targets.
The standard situation is that you are given ground truth labels, that you have learned a binary classifier from those ground truth labels via supervised learning, and that this classifier provides you with a score. Then the ROC score is independent of the threshold you choose for predicting the label of future observations. The ROC curve only depends on the ordering given by the score and on the ground truth variables.
Now to your question: Yes, you can create a binary classifier with any threshold and then report the sensitivity and specificity thereof. This is often done and helps users to choose the classifier they prefer (some might favor high recall, others are more afraid of false positives).
