Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am using a classifier which returns probabilities. To calculate AUC, I am using pROC R-package. The output probabilities from classifier are:

probs=c(0.9865780,
0.9996340,
0.9516880,
0.9337157,
0.9778576,
0.8140116,
0.8971550,
0.8967585,
0.6322902,
0.7497237)

probs shows probability of being in class '1'. As shown, the classifier has classified all of samples in class '1'.

True label vector is:

truel=c(1, 1, 1, 1, 1, 0, 0, 0, 0, 0)

As shown, classifier has misclassified 5 samples. But, AUC is:

pROC::auc(truel, probs)
Area under the curve: 1

Could you please explain to me why it happens?

share|improve this question
    
Related question: stats.stackexchange.com/questions/97395 – Juho Kokkala Mar 10 at 11:39
up vote 15 down vote accepted

The AUC is a measure of the ability to rank examples according to the probability of class membership. Thus if all of the probabilities are above 0.5 you can still have an AUC of one if all of the positive patterns have higher probabilities than all of the negative patterns. In this case there will be a decision threshold that is higher than 0.5, which would give an error rate of zero. Note that because the AUC only measures the ranking of the probabilities, it doesn't tell you if the probabilities are well calibrated (e.g. there is no systematic bias), if calibration of the probabilities is important then look at the cross-entropy metric.

share|improve this answer

The samples weren't "misclassified" at all. The 0 examples are ranked strictly lower than the 1 examples. AUROC is doing exactly what it's defined to do, which is measure the probability that a randomly-selected 1 is ranked higher than a randomly-selected 0. In this sample, this is always true, so it's a probability 1 event.

Tom Fawcett has a great expository article about ROC curves. I'd suggest starting there.

Tom Fawcett. "An Introduction to ROC Analysis." Pattern Recognition Letters. 2005.

share|improve this answer
1  
+1 The Fawcett paper is indeed a very good place to start. – Dikran Marsupial Mar 9 at 19:06

The other answers explain what is happening but I thought a picture might be nice.

You can see that the classes are perfectly separated, so the AUC is 1, but thresholding at 1/2 will produce a misclassification rate of 50%.

probs

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.