# How to calculate the prediction score of a classificator?

I want to compare a given classification algorithm with others via the Area under the (ROC-)curve metric. Unfortunately this algorithm only outputs the values of the respective confusion matrix (TP, FP, TN, FN) and a subset of the predicted positives, but no probability score for any of its predictions.

Confusion Matrix and Statistics
Reference
Prediction TRUE FALSE
TRUE    11    10
FALSE    3   475

Accuracy : 0.9739
95% CI : (0.9559, 0.9861)
No Information Rate : 0.9719
P-Value [Acc > NIR] : 0.46294

Kappa : 0.6156
Mcnemar's Test P-Value : 0.09609

Sensitivity : 0.78571
Specificity : 0.97938
Pos Pred Value : 0.52381
Neg Pred Value : 0.99372
Prevalence : 0.02806
Detection Rate : 0.02204
Detection Prevalence : 0.04208
Balanced Accuracy : 0.88255


When I tried to understand the ROC with examples like this or this, it always requires the prediction score to calculate the AUC and draw the curve. Wikipedia hints, that I should use a probability density function, but I don't know which and how. So, is it even possible to calculate the score and if yes, how?

Thank you guys in advance for your replies.

• You need some sort of quantitative score if you want to make a ROC curve that has more than just one point. If you can show how got this data maybe someone can help you extract the numerical predictions from the model. – Calimo May 7 '18 at 16:43
• Some R classifiers will output probabilities, but since you do not say which one you are using, we can't help you find that. – G5W May 7 '18 at 19:21
• I'm afraid it is not a standard R classifier, but one I implemented out of a research paper. I found out there may be a way to calculate the scoring values. If so, I will write a short answer. – Hendrik May 8 '18 at 6:18