Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've used the R ROCR package to generate an ROC plot. I can see a diagonal in my plot. However, the ROC curve that I see other people generated has only horizontal and vertical lines.

Why would I get a plot like that? Is that possible to have a diagonal line in a legal ROC plot?

ROC curve I see in the literature looks like this:

enter image description here

My generated ROC plot is:

enter image description here

share|improve this question
    
What problem are you trying to solve that made ROC curves have anything to do with it? – Frank Harrell Jan 16 at 18:52
up vote 3 down vote accepted

Assume that you have the following result:

score label
1.000 positive
0.900 negative
0.900 positive
0.900 negative
0.500 negative
0.200 positive

Manually plot the ROC curve for the possible thresholds of 1.0, 0.9, 0.5, 0.2 and you do get a sloped part.

The reason are duplicate scores.

Beware, there are some poor implementations of ROC out there. I've seem some that only sample values (usually you can recognize this because they have very evenly spaced steps), and I've seen implementations that simply sort the data and then take the nth object - ignoring duplicate scores. If the input data is presorted by label, this causes results to be much better than expected. This can be detected by using a data set where all scores are 0 - the only correct result then is the diagonal line and a AuC of 0.5

share|improve this answer
    
@NickCox sorry crappy touchscreens. presorted. If the data was sorted by label at some point, and scores are duplicate, then often all the positives or all the negatives come first. – Anony-Mousse Jan 16 at 20:56

Yes, this is "legal". If the jump from one threshold to the next raises the amount of false positives and false negatives together the result is a diagonal line. Two reasons that might happen:

  1. You have 2 observations with same threshold but with different ground truth
  2. The resolution between 2 thresholds is large enough - in that case you may also check a threshold between the two.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.