How can I decide whether the attached is a correct ROC curve?

The underlying algorithm consists in calculating an ODE system and storing the number of certain cancer cells calculated by this model at the end of the simulation course. These end points are compared with experimental results for different thresholds.

It looks to me as if the model is not performing better then random guess, do I see this correctly?

enter image description here


I rerun the dataset with pROC package - the monotonicity is restored but the rest looks as it was symmetric wrt identity line compared to the original plot (which is worrisome because it means I cannot compute the ROC curve by myself). enter image description here

  • $\begingroup$ If this is indeed a ROC curve then yes, this is a pretty poor model. $\endgroup$ – user2974951 Oct 9 '18 at 8:48
  • 3
    $\begingroup$ Your ROC curve cannot be plotted correctly because an ROC curve has to be monotonically increasing. So we can't conclude anything about the model! $\endgroup$ – Flounderer Oct 9 '18 at 11:29
  • $\begingroup$ @Flounderer the fact that it is not monotonically increasing was one of my worries. And I might have found the bug. The original data frame was defined with TPR/FPR in decreasing order df <- structure(list(TPR = c(1.00000000,0.97159091,...,0.01136364,0.00000000), FPR = c(1.00000000,0.98591549,...,0.02816901,0.00000000)), ...) After reversing the order of my TPRs and FPRs the plot looks correct at least from the monotonicity perspective. $\endgroup$ – emjeh Oct 9 '18 at 12:28
  • $\begingroup$ @Flounderer because I haven't used 'geom_roc' or else, just 'ggplot with geom_line/point' I assume the reversing the order was required. $\endgroup$ – emjeh Oct 9 '18 at 12:31

must ROC curve be concave?

No, a ROC curve can be concave, convex, or a mix of those on different segments (see this question on SO).

How can I decide whether the attached is a correct ROC curve?

A ROC curve always increases monotonically, so the curve you posted is clearly not a ROC curve.

More generally, making a ROC curve is pretty hard and it is easy to get some details and special cases wrong. So if you want to try your own code I would recommend cross-checking with one of the numerous dedicated functions/packages that already exist and have been thoroughly checked.

It looks to me as if the model is not performing better then random guess, do I see this correctly?

It would seem so visually. Try to add a diagonal line and you will see that parts of your curve are above (better than random) and other below (worse than random) the diagonal. However at times even a small signal might be significant, so you want to check for instance with the DeLong test whether the area under the curve AUC is different from 0.5 (H0: AUC = 0.5).

  • $\begingroup$ Which packages would you suggest? I have found only plotROC and .... pROC, I will try that one :) $\endgroup$ – emjeh Oct 9 '18 at 21:07
  • $\begingroup$ I tried this dataframe with pROC 'df <- structure(list(Exp = c("P","N","P","P","P","P","P","N","N","P","P","P","P","P","P","N","P","P","P","P","P"), Pred = c(63.21491,110.82595,55.57196,34.40686,34.16466,53.84683,76.34607,76.34607,94.89890,61.34849,72.92361,93.23873,93.23873,32.52049,93.23873,56.35317,78.07494,76.34607,76.34607,94.89890,63.21491)), .Names = c("Pred","Exp"), row.names = c(NA, -26L),class = "data.frame") blub <- roc(df\$Exp, df$Pred, levels=c("P", "N"))' but get the error: "Error in roc.default(...) : Predictor must be numeric or ordered." What is going wrong? $\endgroup$ – emjeh Oct 9 '18 at 21:56
  • $\begingroup$ Your data.frame is corrupted. I get a warning when I read it, it has extra rows with NA and the columns are wrongly labelled. $\endgroup$ – Calimo Oct 10 '18 at 6:35
  • $\begingroup$ It works, thank you! I have shortened the dataframe due to comment length restrictions and fixed the labels. $\endgroup$ – emjeh Oct 10 '18 at 7:12
  • $\begingroup$ Why does the ROC change (identity line flop, see my comments in Edits section) dependent on the level sequence, is it intended??? Using levels=c("N","P") should be the same as levels=c("P","N") or not? $\endgroup$ – emjeh Oct 10 '18 at 8:35

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.