Area Under the ROC Curve: Comparing identification performance between two values of the same variable

Question

I'm trying to figure out ROC analysis and have the following question:

Assume there is an test instrument that claims to be able to identify cats based on a series of checks (such as: does the animal have four legs? does it eat mice?...).

I want to test this instrument using ROC analysis and a sample of cats and dogs. More specifically, my hypothesis is the following: "The instrument is better at predicting cats than dogs."

How should I go about testing this with the Area Under the Curve (AUC)?

My initial thought was: I compare the AUC with the cats as positives against the AUC with the dogs as positives. But that would mean that I test one instrument and just switch the positives and negatives around. I would be comparing, for example, AUC 0.65 against 1-0.65 (0.35), which feels intuitively wrong. Or is it not?

EDIT: In the latter way of comparing the AUC, I could stick to comparing the AUC for the cats against 0.5, I believe. As my sample is cats and dogs only, a significant deviation from 0.5 means both cats and dogs AUCs are significantly different. Does that make sense?

Answer 1

Your question and proposed solution seems a bit convoluted to me. Also, it wouldn't be very accurate. Your algorithm hasn't been trained to recognize dogs, and it feels weird to try to discriminate dogs with it. After all, it could be giving higher scores/probabilities to dogs than, say, trees, and that wouldn't be a problem as long as it gives even higher scores/probabilities to cats.

The classical approach is to phrase your question slightly differently: can your instrument discriminate between cats and dogs?

In order to answer this question, you build a test set with cats (positive) and dogs (labeled as negative) and calculate the ROC curve. Do you have any discrimination power, is the AUC > 0.5?

This approach is much simpler, easy to test, and I would argue more useful. You can then extend it to any type of negatives, other animals, people, mixed sets etc.

Answer 2

ROC (Receiver operating characteristic) curves allow you to graph the true positive rate (TPR) against the false positive rate (FPR) for a range of thresholds for your test instrument or classifier.

A requirement to analysis using this technique is that your classifier generates a probability score for each instance it sees in the range $[0,1]$. In your toy example, $0$ is "definitely not a cat" and $1$ is "definitely a cat." It then uses a threshold value, $t$, to decide, based on the probability score whether the instance is a cat or not.

To see it in action, imagine your instrument generates the following probability scores for ten instances (5 cats, 5 dogs):

{c, 0.93} {c, 0.81} {c, 0.72} {d, 0.68} {c, 0.64} {d, 0.59} {c, 0.54} {d, 0.49} {d, 0.21} {d, 0.10}

Now you can graph the TPR against the FPR by seeing how these change for different values of $t$.

$t = 0.95$, TPR = 0.0, FPR = 0.0
$t = 0.90$, TPR = 0.2, FPR = 0.0
$t = 0.80$, TPR = 0.4, FPR = 0.0
$t = 0.70$, TPR = 0.6, FPR = 0.0
$t = 0.65$, TPR = 0.6, FPR = 0.2
$t = 0.60$, TPR = 0.8, FPR = 0.2
$t = 0.55$, TPR = 0.8, FPR = 0.4
$t = 0.50$, TPR = 1.0, FPR = 0.4
$t = 0.40$, TPR = 1.0, FPR = 0.6
$t = 0.20$, TPR = 1.0, FPR = 0.8
$t = 0.05$, TPR = 1.0, FPR = 1.0

Graphing these you end up with curves which looks like this:

The Area Under The Curve is then just the integral of this curve here, described on Wikipedia. Here it is easy to calculate it as 0.88. As a rule of thumb, classifiers whose ROC curves are close to reference line (indicating an equal number of true positives and false positives detected) are considered the poorest performing as they show no discriminating ability. Conversely, classifiers whose ROC curve is below the reference line just need to have their decision rule inverted - they can tell "cat" and "not cat" apart, they are simply assigning the wrong label.