I'm trying to figure out ROC analysis and have the following question:

Assume there is an test instrument that claims to be able to identify cats based on a series of checks (such as: does the animal have four legs? does it eat mice?...).

I want to test this instrument using ROC analysis and a sample of cats and dogs. More specifically, my hypothesis is the following: "The instrument is better at predicting cats than dogs."

How should I go about testing this with the Area Under the Curve (AUC)?

My initial thought was: I compare the AUC with the cats as positives against the AUC with the dogs as positives. But that would mean that I test one instrument and just switch the positives and negatives around. I would be comparing, for example, AUC 0.65 against 1-0.65 (0.35), which feels intuitively wrong. Or is it not?

EDIT: In the latter way of comparing the AUC, I could stick to comparing the AUC for the cats against 0.5, I believe. As my sample is cats and dogs only, a significant deviation from 0.5 means both cats and dogs AUCs are significantly different. Does that make sense?

  • $\begingroup$ Have you trained your instrument to recognize dogs? $\endgroup$
    – Calimo
    Aug 4 '17 at 5:21
  • $\begingroup$ No, only cats. But to see how valid it is for cats, I want to see how it performs for dogs and compare. If it is not better for cats than for dogs, I know it is not useful. $\endgroup$
    – Zxifer
    Aug 4 '17 at 6:35

Your question and proposed solution seems a bit convoluted to me. Also, it wouldn't be very accurate. Your algorithm hasn't been trained to recognize dogs, and it feels weird to try to discriminate dogs with it. After all, it could be giving higher scores/probabilities to dogs than, say, trees, and that wouldn't be a problem as long as it gives even higher scores/probabilities to cats.

The classical approach is to phrase your question slightly differently: can your instrument discriminate between cats and dogs?

In order to answer this question, you build a test set with cats (positive) and dogs (labeled as negative) and calculate the ROC curve. Do you have any discrimination power, is the AUC > 0.5?

This approach is much simpler, easy to test, and I would argue more useful. You can then extend it to any type of negatives, other animals, people, mixed sets etc.


ROC (Receiver operating characteristic) curves allow you to graph the true positive rate (TPR) against the false positive rate (FPR) for a range of thresholds for your test instrument or classifier.

A requirement to analysis using this technique is that your classifier generates a probability score for each instance it sees in the range $[0,1]$. In your toy example, $0$ is "definitely not a cat" and $1$ is "definitely a cat." It then uses a threshold value, $t$, to decide, based on the probability score whether the instance is a cat or not.

To see it in action, imagine your instrument generates the following probability scores for ten instances (5 cats, 5 dogs):

{c, 0.93} {c, 0.81} {c, 0.72} {d, 0.68} {c, 0.64} {d, 0.59} {c, 0.54} {d, 0.49} {d, 0.21} {d, 0.10}

Now you can graph the TPR against the FPR by seeing how these change for different values of $t$.

$t = 0.95$, TPR = 0.0, FPR = 0.0
$t = 0.90$, TPR = 0.2, FPR = 0.0
$t = 0.80$, TPR = 0.4, FPR = 0.0
$t = 0.70$, TPR = 0.6, FPR = 0.0
$t = 0.65$, TPR = 0.6, FPR = 0.2
$t = 0.60$, TPR = 0.8, FPR = 0.2
$t = 0.55$, TPR = 0.8, FPR = 0.4
$t = 0.50$, TPR = 1.0, FPR = 0.4
$t = 0.40$, TPR = 1.0, FPR = 0.6
$t = 0.20$, TPR = 1.0, FPR = 0.8
$t = 0.05$, TPR = 1.0, FPR = 1.0

Graphing these you end up with curves which looks like this: The ROC curve corresponding to the data points shown above, along with a reference line showing "no discriminating ability."

The Area Under The Curve is then just the integral of this curve here, described on Wikipedia. Here it is easy to calculate it as 0.88. As a rule of thumb, classifiers whose ROC curves are close to reference line (indicating an equal number of true positives and false positives detected) are considered the poorest performing as they show no discriminating ability. Conversely, classifiers whose ROC curve is below the reference line just need to have their decision rule inverted - they can tell "cat" and "not cat" apart, they are simply assigning the wrong label.

  • $\begingroup$ Thank you for your answer. I understand what you explained. Although my question was more about how to approach my specific hypothesis "The instrument is better at predicting cats than dogs." using ROC/AUC. $\endgroup$
    – Zxifer
    Aug 3 '17 at 19:31
  • $\begingroup$ @Zxifer If I understand correctly, then, you want to compare between two ROC curves: one representing the classification of cats and the other representing the classification of dogs. I would suggest you look at Mason and Graham's article which demonstrates "that the ROC area is equivalent to the Mann–Whitney U-statistic testing the significance of forecast event probabilities for cases where events actually occurred with those where events did not occur." $\endgroup$
    – user77876
    Aug 3 '17 at 21:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.