3
$\begingroup$

Why do I see my ROC curve crossing the line from (0,0) to (1,1) (i.e. the 0-1 line)?

I have the following test data as a tab-separated testdata.txt file. Running my R code (given below) multiple times, I get a few ROC curves that cross the 0-1 line multiple times. What may be the possible explanation for this, other than not enough data?

Thanks.

testdata.txt

pred1   pred2   pred3   response
25  3.8 13.4    0
29  5.3 23.5    1
30  4.3 14.1    1
29  3.8 16.1    0
26  5.7 22.9    0
20  4   13.6    0
28  3.2 14.8    0
23  5.1 16.9    0
24  4.3 15.2    0
25  5   15.2    0
29  4.1 13.7    0
24  3.7 14.7    0
29  4.7 13.8    0
28  3.9 14.2    0
25  4.6 15.4    0
24  3.8 13.9    0
33  4.3 17.1    0
33  5   16.6    1
28  4.6 15.2    0
29  4.5 15.6    0
26  4.1 14.3    0
31  4.4 12.6    0
24  4.7 15.2    0
21  5   16.1    0
24  4.1 13.3    0
29  4.1 17.7    1
29  4.2 18.4    1
27  4.5 17.3    0
27  4.4 16.4    1
28  4.5 17.6    1
29  4.7 17.8    1
25  4.1 15.2    0
29  4   13.3    0
29  3.9 13      1
27  3.9 17.9    0
25  3.2 13.6    0
22  4.8 13.4    0
26  5.5 14.2    0
24  4   16.6    0
38  3.2 14.8    0
33  3.8 13.6    1
24  3.1 16.5    1
23  4.8 14.5    0
27  5.4 13.5    0
27  4.1 12.8    1
30  3.9 13.1    0
24  5.4 13      0
25  4   12.3    0
28  4   16.5    0
30  4.6 13.5    0
18  3.6 13.7    0
24  3.2 13      0
27  4   14.3    0
33  3.6 17.1    0
24  3.2 15.5    0
24  3.2 13.3    0
24  4   14.2    0
26  4.2 14.3    0
23  3.2 16.9    0
16  3.4 17.1    0
21  4.3 21.1    0
27  4.5 13.5    0
29  4.1 13.5    0
27  3.9 14.1    0
27  4.1 13.6    0
27  4.6 12.7    0
23  4   15.2    0
29  3.7 15.1    0
33  3.6 17.1    1
16  5   16.6    0
21  5.4 16.7    0
27  4   11.5    0
29  3.7 12.7    0
26  3.7 11.7    0
16  4.3 16.6    0
25  4.3 13.5    0
24  3.8 12.9    0
33  3.2 14.1    0
24  3.4 14.2    0
26  4.1 13.9    0
26  5.3 14.4    0
16  4.3 21.1    0
20  5.4 17.2    0
29  4.1 13.1    1
32  3.1 13.7    0
31  3.8 12.8    0
29  4.8 16.7    0
33  4   14.7    0
16  5   17.1    0
29  4.6 13.2    0
23  3.4 12.7    0
26  4.5 12.2    0
25  4   13.4    0
31  3.6 13.5    0
32  5.3 13.9    1
33  4.3 16.6    0
29  4.1 15.3    0
18  4.2 14.9    0
24  5.9 16.7    0
25  4.3 11.8    0
29  4.3 13.7    0
27  5.7 13.6    0
29  4   12.4    0
24  4.5 13.8    0
27  3.8 14.1    0
34  4.2 12.4    0
30  4.3 13.1    0
26  3.4 15.9    0
22  4.3 15.6    1
33  4.2 13.2    0
25  3.2 13.4    0
28  4.6 15.5    0
24  4.8 14.6    0
24  3.9 12      0
29  4.9 14.7    0
28  4.5 13.5    0
25  5   15.2    0
28  4.8 14      0
35  3   15.1    0
27  4.5 12.8    0
31  4.6 15.6    0
22  4.7 14.3    0
25  3.9 13.6    0
28  5.6 14.7    0
21  4.3 19.9    1
31  3.8 16.1    1
26  4   15      1
21  4.8 14.4    0
30  4.3 14.7    0
25  5.3 12.1    0
30  4.8 13      0
29  3.9 15.1    0
32  4.4 14.7    1
24  4.3 14.2    0
29  4   14.2    0
29  4.3 12.6    0
23  3.7 17.5    0
29  4.2 14.8    0
29  4.8 15.2    0
31  6.1 13.6    0
27  4.3 13.1    0
27  4.4 12.3    0
28  5.1 13.9    1
32  4.2 11.9    0
30  4.3 12.5    0
26  3.7 12.2    0
28  4   12.5    0
29  2.8 13.5    0
30  3.6 12.2    0
29  4.1 13.5    0
27  5.2 16.5    0
20  6.4 16.7    0
29  4.6 12.1    0
24  4   14.2    0
22  3.8 13.3    0
34  4.6 13.4    1
31  4.5 13.6    0
28  4.9 12.5    0
33  4   12.6    0
29  4   13.9    0
29  4   12.7    0
28  3.8 12.5    0
27  4.2 14.2    0
29  3.8 15.2    0
26  3.8 13.5    0
27  5.8 13.8    0
22  3.8 11.7    0
28  3.7 13.2    0
25  4.1 13.9    0
29  4.1 14.1    0
13  4.3 19.3    0
27  4.5 18.3    0
24  4.6 13.8    0
37  3.8 12.3    0
25  4.8 11.9    0
30  4.1 12.5    0
23  3.7 13      0
26  3.8 14.1    0
34  4.1 12.9    0
24  4.5 16.9    0
25  3.7 12.7    0
30  4   12.7    0
25  5   12.3    1
29  3.9 14.2    0
25  3.9 13.6    0
27  4.5 12.8    0
29  4.1 13.6    0
28  4   13.4    0
28  4.1 13.9    0
27  3.5 13.9    0
26  2.7 18.8    1
30  4.6 12  0
25  4   13.6    0
26  4.9 14.7    0
24  4.1 13.2    0
26  4   13.2    0
16  4.3 15      0
33  5.3 15.7    0
24  4.8 15.5    0
29  3.2 13.6    0
30  4.4 12.6    0
28  3.8 13.5    1
24  4.3 13.3    0
33  4.2 13.8    0
30  4.2 13.2    0
26  3.6 12.5    0
24  4.1 13.5    1
16  3.1 13      0
24  3.4 17.6    0
24  3.2 17.6    0
24  5.4 16.9    0
29  4   16.2    0
25  4.2 13.2    0
21  6   19.6    1
25  4.3 13.5    0
28  3.7 12.2    0
27  4.1 14      0
16  4.1 19.7    0
29  4.4 14.6    0
32  4.1 16.9    1
29  3.6 15.6    1
30  4   15.7    0
23  4.4 14.4    0
30  2.9 12.8    0
27  4.5 11.8    0
24  4   17.4    0
30  4   13.1    0
30  4   12      0
24  5   14.2    0
24  4.5 13      0
26  3.7 16.1    0
27  3.2 15.1    1
27  3.5 12      0
30  4.6 13      0
24  4   14.9    0
31  3.9 13.8    0
29  4.1 14.6    0
25  6.4 19.6    0
25  4.1 14.8    0
33  3.6 15.4    0
32  3.9 14.4    0
25  4.1 12.1    0
24  3.7 13.3    0
23  4.1 14.3    0
28  4.7 13.5    0
33  4.1 15      0
26  3.8 12.4    0
32  4.2 11.7    0
23  3.7 13.8    0
30  3.4 12      0
32  3.6 11.8    1
26  4.1 12.4    0
24  4.3 12.5    0
24  4.5 15.5    1
26  4.7 13.5    0
26  4.4 13.3    0
33  4.3 18.3    0
28  4.2 13.3    0
23  4.5 23.9    0
25  3.8 12.4    0
22  5.5 15.5    1
30  3.5 12.8    0
30  4.1 11.8    0
29  4.5 12.1    0
28  3.8 13.6    0
25  3.9 12.9    0
29  4.2 13.2    0
29  3.8 13.8    0
25  4.1 12.7    0
27  4.4 12.2    0
24  3.4 13.9    1
24  4.1 16.4    1
21  5.9 22.7    0
21  5   16.6    0
26  4.7 13.4    0
24  3.9 12.7    0
27  4.2 12.2    0
23  4   13.5    0
24  3.8 14.2    0
22  3.9 12.6    1
30  5.1 13.7    0
25  3.9 14.2    0
25  4.4 12.1    0
23  4.1 13.8    0
30  4   12.9    0
27  4.6 21      0
33  3.1 17.1    0
26  3.1 13.6    0
26  5.7 17.1    0
28  4   14.2    1
26  4.3 12.2    0
25  4.1 13.3    0
28  3.4 14.6    0
24  3.9 13.6    0
27  3.9 12.5    0
24  3.4 13.9    0
27  4   12.5    0
24  4.3 13.9    0
25  4.1 13.4    0
24  5.3 19.6    0
31  4.9 13.3    0
25  4.2 13.1    0
21  3.7 13.9    0
26  4.1 11.7    0
24  4   13.3    0
27  3.7 13.2    0
26  4.6 13      0
24  4   13.6    1
25  3.7 15.7    0
26  4.3 13      1
23  3.7 15      0
28  4.1 14.4    0
22  5.4 13.7    0
27  5.3 13.8    0
25  4.2 13.1    0
26  4.3 13.9    0
29  4.8 16      0
23  3.4 16.7    0
27  4.2 15.7    0
29  3.8 12.1    0
27  4.2 13.9    0
24  3.9 13.5    0
27  5.7 17.8    0
29  4.1 14.3    0
29  5   13.2    0
34  4.5 11.8    0
26  5.7 17.1    0
28  3.8 18.6    0
29  4.5 14.2    0
31  3.8 21.4    0
30  4.7 13.7    0
27  4   18.6    0
24  4   14.7    0
29  4   15.2    0
26  3.4 11.9    1
27  4.3 13.4    0
30  4.6 13.7    0
29  4.2 13.1    0
22  4.3 13.2    0
28  4.4 13.1    0
21  3.7 14.2    0
25  3.8 17.4    0
28  4   18      0
26  3.6 17.3    0
24  4.3 13.3    0
32  4   12.1    1
31  4.2 12.5    1
38  3.1 19.6    1
28  4.3 13.1    1
22  4.5 17.1    0
24  3.2 13.8    0
30  3.9 13.7    0
26  4.2 12.2    0
25  3.2 17.1    0
26  4.5 13.2    0
31  4   13.6    0
33  4.3 17.1    0
30  4.1 13      0
21  4.3 14.7    1
25  3.9 13.3    0
16  3.2 13.6    0
26  4.3 16.6    0
21  4.3 16.6    0
28  3.8 11.6    0
24  4.4 12.8    0
30  4   12.4    0
30  5.6 14.9    0
26  4.5 13.6    0
27  4.3 11.8    0
27  4.2 11.5    0
26  3.6 13.6    0
24  3.1 19.3    0
24  4   14.8    0
25  3.8 12      0
23  4.2 12.8    0
23  4.1 14.3    0
30  4.2 12.4    0
28  4   13      0
29  4.7 13.2    1
23  4.9 14.7    0
22  5.3 14.9    1
25  5.5 15.1    1
34  4.3 14.7    0
26  4.1 12.3    0
24  4   16.2    0
29  4.5 12.5    0
21  4.3 17.1    0
30  4.2 12.1    0
25  4.1 13.6    1
26  4   14.2    0
33  3.9 11.7    0
24  3.2 13.2    0
25  3.6 12.5    0
27  4.3 13.5    0
25  3.4 15.2    0
29  4.3 13.4    1
24  4.2 13.5    0
21  5.9 19.6    0
26  3.4 25.6    0
26  4.3 15.4    0
29  4.5 15.6    0
28  4.1 14.6    0
28  4.4 16.9    0
21  3.9 14.2    0
24  3.4 13.9    0
25  4.3 13.5    0
33  5.5 19.9    0
29  3.9 16.9    0
24  3.4 14      0
24  3.9 16.7    0
23  4   13.4    0
30  4.6 15.1    0
32  4   13.3    0
16  3.2 19.6    0
29  4.5 16.6    1
28  5   14.8    0
29  4.1 15.3    0
25  4.2 14      0
26  3.5 11.9    0
30  4   12.2    1
25  4   13.8    0
24  3.4 14.2    0
25  3.4 12.4    0
24  4.3 14.5    0
25  4   13      1
35  4.3 14.9    0
24  5.3 14.7    0
25  3.4 13.6    0
22  3.8 15.6    0
24  4.2 14.3    0
33  8.3 21.2    0
30  4.2 13.8    0
25  3.8 13.1    0
32  3.5 14.5    0
29  3.7 11.7    0
23  4   14      1
24  4   12      0
32  4.2 13.8    0
24  4   12.1    0

R code

myData = read.table("testdata.txt", sep="\t", header=TRUE)
nr = length(myData[,1])
nr
testidx = sort(sample(1:nr, 0.3*nr, replace=FALSE))
dataTraining <- tmpData[-testidx,]
dataTesting <- tmpData[testidx,]

(fmla <- response ~ pred1 + pred2 +pred3)
model <- glm(formula = fmla, family=binomial(), na.action=na.exclude, data=dataTraining )
summary(model)

yTest = dataTesting[,'response']  #actual values on testing data
yhatTest = predict(model, newdata=dataTesting, type = "response")    #values predicted by the model on testing data
plot.roc(yTest, yhatTest,  col = "red", print.auc = TRUE, print.auc.y = 0.05)

One such ROC curve is given below:

enter image description here

Improve this question
$\endgroup$

2 Answers 2

Reset to default
7
$\begingroup$

The ROC curve looks pretty similar to random guessing to me. Occasionally the model get lucky, but it doesn't stay lucky.

/edit to expand a little bit. Lets say we have 3 completely random classifiers, that are unrelated to our outcome of interest. In R, we can do this as follows:

library(caTools)
set.seed(42)
rows <- 100
cols <- 3

predictors <- matrix(runif(rows*cols), ncol=cols)
obs <- sample(0:1, rows, replace=TRUE)

colAUC(predictors, obs, plotROC=TRUE)
lines(0:1, 0:1)

ROC1

You can see, with a small number of examples, these random classifiers occasionally get lucky (or unlucky), and fall above or below the .50 line. These fluctuations are random and as we increase the number of observations we see that the 3 ROC curves get closer to the line:

library(caTools)
    set.seed(42)
    rows <- 10000
    cols <- 3

    predictors <- matrix(runif(rows*cols), ncol=cols)
    obs <- sample(0:1, rows, replace=TRUE)

    colAUC(predictors, obs, plotROC=TRUE)
    lines(0:1, 0:1)

ROC 2

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks. That true for the data. But why the curve is alternating going sometimes above and below. I am expecting a little above or little below the nondiscriminating line. $\endgroup$
    – user1140126
    Commented Jun 19, 2013 at 13:58
  • $\begingroup$ @user1140126 It looks like you have ~ 16 instances of the positive class. At the points where these instances fall on your predicted probabilities, you will get a vertical jump in the ROC curve. If you have several instances with similar probabilities, you will get a bunch of vertical jumps, which may cross the line. $\endgroup$
    – Zach
    Commented Jun 19, 2013 at 16:49
0
$\begingroup$

I had the same problem. As pointed above by Zach, this means your classifier has similar performances to a random classifier.

However, there can also be another bottleneck: your ROC algorithm. In my case, my computation of the ROC curve was wrong, and thus it was not my classifier which was random, but the way the ROC curve was computed, leading to this kind of result.

To avoid this problem, you should use a few sanity checks. A good one I have used was to compute:

  1. the total number of real Negative examples (eg: Y==0)
  2. the total number of real Positive examples (eg: Y==1)
  3. at (1.0, 1.0) of the ROC curve, the total number of Negative examples predicted by your classifier (eg: False Positive + True Negative)
  4. at (1.0, 1.0) of the ROC curve, the total number of Positive examples predicted by your classifier (eg: True Positive + False Negative)

You should then compare 1 with 3, and 2 with 4. If they are not equal (1 == 3 and 2 == 4), then your ROC curve is wrong.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.