Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm reading Fawcett's 2004 paper on ROC graphs for machine learning algorithms, which can be found here.

On page 7-8 he shows a simple ROC example and makes some interpretations that I don't understand. Below is the ROC graph:

example ROC graph

And here is what he wrote:

Although the test set is very small, we can make some tentative observations about the classifier. It appears to perform better in the more conservative region of the graph; the ROC point at (0.1,0.5) produces its highest accuracy (70%). This is equivalent to saying that the classifier is better at identifying likely positives than at identifying likely negatives. Note also that the classifier’s best accuracy occurs at a threshold of ≥ .54, rather than at ≥ .5 as we might expect with a balanced distribution.

I don't understand how he derived his interpretations.

  1. the ROC point at (0.1,0.5) produces its highest accuracy (70%) How is the highest accuracy of 70% for point (0.1, 0.5) found from that graph, and how do we know it's the highest accuracy?

  2. This is equivalent to saying that the classifier is better at identifying likely positives than at identifying likely negatives. I don't see how that interpretation is determined.

  3. Note also that the classifier’s best accuracy occurs at a threshold of ≥ .54 How was this found?

  4. rather than at ≥ .5 as we might expect with a balanced distribution Why would we expect that?

Thank you for any help.

share|improve this question
add comment

2 Answers 2

up vote 4 down vote accepted
  1. The accuracy is the ratio of correct results (true positives and true negatives) to total number of tests. If you classify the first six results in the table are "positive" and the rest are "negative", that gives 5 true positives and 9 true negatives. The accuracy is 14/20, which is higher than any other threshold point on the curve.

  2. If you use this classification rule, 5/6 of the data you classify as "positive" are correct, but only 9/14 of those you classify as "negative" are really negative. Seeing an observation classified as "positive" is more trustworthy than seeing a "negative" classification.

  3. A decision rule that classifies scores $\geq .54$ as "positive" is the threshold that chooses the top 6 scores in this sample.

  4. A balanced distribution of the score would imply that true negatives get high scores just as often as true positives get low scores. And in this example it looks like we're assuming that the proportions are .50-.50 positive and negative.

share|improve this answer
    
Thanks. For 1, how do you know to only look at the first six results to be positive? Why not the first 8 or 9? For 2, how do you get "The accuracy is 14/20, which is higher than any other threshold point on the curve"? What is a threshold point? Is it when you go from upwards to rightwards? And do you have to try out all such points to find the max accuracy? –  stackoverflowuser2010 Feb 17 at 22:57
    
I'm referring to a threshold for the scores in the 3rd and 6th columns of the table. A classifier means picking a number and calling all scores higher than that "positive" and all lower scores "negative". Numbers 7 and 8 are truly negative, so if you lower the threshold to class them as "positive" you'll get two fewer correct decisions. A threshold of .5 gives 5 true positive and 5 true negative for accuracy 10/20=.5, not great. –  vafisher Feb 17 at 23:01
add comment

My understanding is that these are not mathematical assessments. One way to get a best point on the curve is to find the one most protruding in the 10:45 clock direction. But that's assuming you care about fpr equally as tpr, which is not always the case. All subsequent statements are based on this choice.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.