I was comparing the performance of various models on a binary classification task I have been working on. When I plot the ROC curve for Gradient Boosting, this is what I get:

Gradient Boosting ROC

However, when I apply a random forest model to the same data, I get an incomplete ROC curve, or certainly a very fishy one.

Random Forest ROC

The Random Forest model's performance just seems to good to be true, is it really the case it is doing that well? When I check the probabilities, a lot of them have the value 1. I suspect an information leak, but then not sure how pandas' train_test_split could be causing that.

  • $\begingroup$ Is your output variable (or some derivative thereof) included in your input variables? You could also check k-fold cross validation. Otherwise I'd say: Congratulations =) $\endgroup$ – such Nov 2 '17 at 11:23
  • $\begingroup$ Doesn't seem to be the case that the output is included in the input variables. I did try k-fold cross validation and got a mean accuracy (recall and precision as well) hovering around 97% percent. Confusion matrix also points to a low number of false positives and false negatives. $\endgroup$ – Aayush Chadha Nov 2 '17 at 11:27
  • $\begingroup$ This just look like you forgot to anchor the curve to (0, 0) (where threshold = infinity). Did you calculate the ROC curve yourself or did you use an existing function/package to do that? This usually results in AUC that are underestimated (here, not by much, but still if the last FPR > 0). Besides, why would it be too good to be true? $\endgroup$ – Calimo Nov 2 '17 at 11:40
  • $\begingroup$ I used sklearn to calculate it. On some hold out test sets, I get a full ROC curve, so not sure about anchoring being an issue. I felt it too good to be true because this is pretty much using an out of the box model without any optimisation. Though looking at some of the variables more closely, it does seem it might be easy for the model to achieve such high performance. $\endgroup$ – Aayush Chadha Nov 2 '17 at 11:49

By definition, ROC curves go from (0,0) to (1,1). You're missing the point at (0,0).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.