2
$\begingroup$

I am using ROC curves for multi-label classification. I have a classifier that produces a score for each label, say a Logistic Regression that produces a probability. I understand that an ROC curve is parameterized by a discrimination threshold and assigns to a class the observations where a class with the highest probability is above the threshold.

If so, imagine these predictions for 5 observations with labels A or B:

Observation #  Label  Prob(A)  Prob(B)
            1    A     0.9        0.1
            2    A    0.51       0.49
            3    B    0.51       0.49
            4    A    0.49       0.51
            5    B    0.49       0.51

The first observation is a freebie. With a discrimination threshold of 0.9, we assign that observation correctly and no observation incorrectly. So True Positives are 1 and all others are zero (True Negatives, False Positives, False Negatives). The True Positive Rate is 1 and the False Positive Rate is 0, which is the ideal point at the top left in an ROC curve. We never see that point in an ROC curve, so I suspect my reasoning is wrong, or my concept of True/False Positive Rates is wrong.

Another possibility is to assign only observations with a probability above a threshold to the most likely class, and all others to the negative class. But that approach lumps together an observation that we are sure is in the negative class and one that we're not sure is in the positive class. A consequence is that it is not invariant under re-labeling (positive to negative and vice-versa).

How exactly does an ROC curve use the discrimination threshold?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I'm confused with your example, what is label 1 (index 1)? Is A the positive class? If so with threshold 0.9 you will missclassify observations 2 and 4...? $\endgroup$
    – Calimo
    Apr 25 at 7:39
  • $\begingroup$ @Calimo I edited the question. The positive class can be A or B, since with more labels I will draw an ROC curve for each label. With a threshold of 0.9, my understanding is that we only classify observation 1, and the other observations are not assigned. Am I wrong? $\endgroup$
    – miguelmorin
    Apr 26 at 7:35
  • $\begingroup$ From your comment, I understand that I am wrong and that at a given threshold, we assign observations with a probability above it to their predicted classes (observation 1 to class A), and all other observations to the negative class (observations 2-5 to class B). Is that right? $\endgroup$
    – miguelmorin
    Apr 26 at 7:46
  • 1
    $\begingroup$ Yes it's right, we do that. So in your example we assign 1 to A and 2-5 to B. So 2 and 4 are wrongly assigned to B. That's not a perfect classification, and that's why the AUC is < 1. $\endgroup$
    – Calimo
    Apr 27 at 6:23

1 Answer 1

Reset to default
1
$\begingroup$

You seem to have a few misunderstandings about ROC curves.

I am using ROC curves for multi-label classification.

ROC curve are tools to assess the discrimination ability of binary classifiers. Some extensions exist for different types of problems such as multi class or multi label classification, but they are not ROC curves strictly speaking.

an ROC curve is parameterized by a discrimination threshold

A ROC curve is parameterized over all possible discrimination thresholds between $-\infty$ and $+\infty$.

With a discrimination threshold of 0.9, we assign that observation correctly and no observation incorrectly.

With a threshold of 0.9, we indeed (correctly) assign observation 1 to the positive predicted class.

All other observations are assigned to the negative predicted class. Because observations 2-5 are < 0.9, we assign them to the negative predicted class. As a result, observations 2 and 4, which should be positive, are misclassified as negatives, and decrease the True Positive Rate (sensitivity) and the AUC.

Because ROC curve is designed for binary classification problems, there is no such thing as "unassigned". If things are not positive, they are negative. If this assumption is not appropriate for your problem, then you're not having a binary classification problem, and ROC curves may be the wrong tool for you.

The True Positive Rate is 1 and the False Positive Rate is 0, which is the ideal point at the top left in an ROC curve. We never see that point in an ROC curve

This is wrong, this point is seen as soon as you have a perfect classifier. This might be hard to achieve in your field or for your problem, but it definitely exists.

How exactly does an ROC curve use the discrimination threshold?

I refer you to this CV question: What does AUC stand for and what is it?, which should answer this part of your question.

Improve this answer
$\endgroup$
6
  • $\begingroup$ So in my example, if we relabel class A as the negative class, at thresholds between 0.52 and 0.9, all observations are assigned to the negative class, right? This implies that the ROC is not invariant to class re-labeling and that ROC is not the right tool. What do you suggest for gauging algorithms that predict multiple classes? $\endgroup$
    – miguelmorin
    Apr 27 at 13:54
  • 1
    $\begingroup$ "at thresholds between 0.52 and 0.9, all observations are assigned to the negative class, right": except the first observation which is going to be positive. $\endgroup$
    – Calimo
    Apr 27 at 18:43
  • $\begingroup$ I don't understand your first sentence: "When B is the positive class, at threshold 0.52 it is predicted to be negative". What is "it" referring to, what is predicted to be negative? $\endgroup$
    – Calimo
    Apr 30 at 18:35
  • $\begingroup$ When B is the positive class, at threshold 0.52 observation 1 is predicted to be in class A, and so assigned to the negative class. The other four observations are below the threshold and also assigned to the negative class, or A. Is that correct? $\endgroup$
    – miguelmorin
    May 2 at 7:17
  • 1
    $\begingroup$ If you call B positive, at threshold 0.52 observation 1 (0.9) is above the threshold, therefore predicted to be positive, IE in class B. There are no labels in ROC analysis, only binary classes which you can call the way you want, A or B, positive or negative, patient or control, true or false. Don't let that confuse you. Maybe it's easier if you stop thinking of A and B and simply focus on positive and negative. $\endgroup$
    – Calimo
    May 2 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.