0
$\begingroup$

I was comparing the performance of various models on a binary classification task I have been working on. When I plot the ROC curve for Gradient Boosting, this is what I get:

Gradient Boosting ROC

However, when I apply a random forest model to the same data, I get an incomplete ROC curve, or certainly a very fishy one.

Random Forest ROC

The Random Forest model's performance just seems to good to be true, is it really the case it is doing that well? When I check the probabilities, a lot of them have the value 1. I suspect an information leak, but then not sure how pandas' train_test_split could be causing that.

$\endgroup$
4
  • $\begingroup$ Is your output variable (or some derivative thereof) included in your input variables? You could also check k-fold cross validation. Otherwise I'd say: Congratulations =) $\endgroup$
    – such
    Nov 2, 2017 at 11:23
  • $\begingroup$ Doesn't seem to be the case that the output is included in the input variables. I did try k-fold cross validation and got a mean accuracy (recall and precision as well) hovering around 97% percent. Confusion matrix also points to a low number of false positives and false negatives. $\endgroup$ Nov 2, 2017 at 11:27
  • $\begingroup$ This just look like you forgot to anchor the curve to (0, 0) (where threshold = infinity). Did you calculate the ROC curve yourself or did you use an existing function/package to do that? This usually results in AUC that are underestimated (here, not by much, but still if the last FPR > 0). Besides, why would it be too good to be true? $\endgroup$
    – Calimo
    Nov 2, 2017 at 11:40
  • $\begingroup$ I used sklearn to calculate it. On some hold out test sets, I get a full ROC curve, so not sure about anchoring being an issue. I felt it too good to be true because this is pretty much using an out of the box model without any optimisation. Though looking at some of the variables more closely, it does seem it might be easy for the model to achieve such high performance. $\endgroup$ Nov 2, 2017 at 11:49

1 Answer 1

1
$\begingroup$

By definition, ROC curves go from (0,0) to (1,1). You're missing the point at (0,0).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.