2
$\begingroup$

I am trying to compare several models of classification tree using the ROCR package however, the x-axis in this package correspond to the rate of positive prediction whereas in every blog/forum where I searched it corresponds to the population%.

Therefore I don't have any idea how to interpret the curves I have.

You can find below the code I wrote for each model and the plots :

for (i in 1:nfold) {
  tree.result[[i]]$roc$prediction <- prediction(tree.result[[i]]$data$predProb, tree.result[[i]]$data$real)
  tree.result[[i]]$roc$lift       <- performance(tree.result[[i]]$roc$prediction, "lift", "rpp")
}
# nfold = number of split in the data/number of models

plot(tree.result$model.1$roc$lift, col="blue")
plot(tree.result$model.2$roc$lift, add=TRUE,col="red")
plot(tree.result$model.3$roc$lift, add=TRUE,col="green")

Result :

enter image description here

Do you know how to interpret this curve ? Is it exactly the same principle than with population% in abscissa?

Improve this question
$\endgroup$
2
$\begingroup$

The underlying idea of a lift chart is really the same, whether using a population (generally a population response to some marketing effort) or a prediction success rate.

enter image description here

In this case, we're looking at the improvement of predictions as a function of the unpredicted values. For example, at the level where a naive effort could produce a 20% rate of positive prediction, the model you have charted in blue would produce about a 1.23 multiple of that, or approximately 24.6%.

This makes sense visually, as the naive prediction rate trends toward 1, the possible improvement multiple declines sharply as well and in this case the difference between the models contract and necessarily converge at rpp = 1.0

The correct interpretation of this lift chart is that the model plotted in red gives the greatest predictive lift.

That is, at least over the visble range of ~.27 to ~.67.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you, your answer is very clear ! As your answer suggested, I plotted the lines in a different order to see above ~1.28 and got a really weird result... ![valid XHTML][graph]. I interpret it as for the very first positive predicition, the effect of model blue which was red in the previous chart and the green one have a tremendous effect. Am I right? [graph]: imgur.com/A9ZWajo $\endgroup$ – Yohan Obadia Sep 15 '15 at 23:51
  • $\begingroup$ So, based on the algorithm's behavior as it approaches 1, what do you think is going on there as it approaches 0? $\endgroup$ – Thomas Cleberg Sep 16 '15 at 0:05
  • $\begingroup$ When we only have to predict a few positive values, the performances of the green and red models are way better and decline really fast as we get slightly away from 0. Hence I would infer that a tree leaf that does not have a big weight is very good in predicting positive values. $\endgroup$ – Yohan Obadia Sep 16 '15 at 9:39
  • $\begingroup$ Without knowing more about your model, I'd say that it's likely that the blue and green models correctly get 1 (or some small number of) predictions correct that the red model doesn't and the naive method never would. $\endgroup$ – Thomas Cleberg Sep 16 '15 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.