2
$\begingroup$

I have two classifiers which try to classify the same data sets. In order to check the efficiency of the classifiers I intend to plot the curves and calculate the AUC value. The concern is that one of the classifiers always produce positive values as the scores (actually it is the p value) but the other classifier produces both positive and negative values as the scores(it has its own producing score method rather than producing pvalues). Should I take the absolute values for the second scores when plotting the roc curves and calculating the AUC or I can use scores as they are (both positive and negative). When I make the absolute values for the second classifier to plot the curves the AUC amount decreases.

Thank you in advance.

Improve this question
$\endgroup$
3
  • $\begingroup$ ROC is the curve of sensitivity and specificity, which is the conditional probability. And AUC is the area under the curve. What do you mean by the negative score? It seems that all these are positive by definition. $\endgroup$
    – Vincent
    Sep 17, 2013 at 9:13
  • $\begingroup$ @Vincent many classifiers output negative scores. SVM is a popular example, where the predicted class label $y \in \{0,1\}$ is then computed as sign(score). A score can have any value and is certainly not necessarily a probability. $\endgroup$
    – Marc Claesen
    Sep 17, 2013 at 9:14
  • $\begingroup$ If different classifiers have different definition or scale about the score, they can not be compared directly unless it is known how they are calculated in different algorithm. $\endgroup$
    – Vincent
    Sep 17, 2013 at 10:19

2 Answers 2

Reset to default
3
$\begingroup$

Always use the scores the way they come out of the classifier. If you take absolute values you are basically changing the classifier's ranking and you will obtain an erroneous ROC curve.

The values of scores across classifiers are entirely irrelevant: the only thing that matters to plot ROC curves is the ranking produced by each classifier, which is based on the scores. In ROC analysis you only compare rankings, irrelevant of what the scores may have been.

If you insist on transforming scores (which is useless), whatever transformation you do must keep the original ranking intact. For example, you could scale the scores with a positive constant or add an arbitrary (but constant!) value to them.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Do you think it is unfair to compare classifiers which produce different types of scores based on ROC their AUC values? $\endgroup$
    – user30314
    Sep 17, 2013 at 12:01
  • 1
    $\begingroup$ No, it is not unfair. That's what ROC analysis is for. ROC compares the ranking capabilities of classifiers. The way they rank is irrelevant (e.g. the actual score values). It's perfectly valid to use ROC analysis to compare, say, logistic regression with SVM. $\endgroup$
    – Marc Claesen
    Sep 17, 2013 at 12:33
0
$\begingroup$

A classifier often involves an internal (hidden) value which is compared against a threshold. Effectively in a ROC curve each such value is treated as a threshold to get each new segment of the curve.

If you are worried about the different range of values, and their unintepretability, it is nice to convert them to a probability. You can do that by reading off new numbers from the ROC curve as each point corresponds to a particular number of positives and negatives that are above threshold. In fact you can derive both a cumulative probability and an incremental probability that relates to the slope at a returned value (versus the slope of the chance line).

There are a couple of papers around that go into detail about this...

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.