R - Lift chart analysis & classification tree - ROCR

Question

I am trying to compare several models of classification tree using the ROCR package however, the x-axis in this package correspond to the rate of positive prediction whereas in every blog/forum where I searched it corresponds to the population%.

Therefore I don't have any idea how to interpret the curves I have.

You can find below the code I wrote for each model and the plots :

for (i in 1:nfold) {
  tree.result[[i]]$roc$prediction <- prediction(tree.result[[i]]$data$predProb, tree.result[[i]]$data$real)
  tree.result[[i]]$roc$lift       <- performance(tree.result[[i]]$roc$prediction, "lift", "rpp")
}
# nfold = number of split in the data/number of models

plot(tree.result$model.1$roc$lift, col="blue")
plot(tree.result$model.2$roc$lift, add=TRUE,col="red")
plot(tree.result$model.3$roc$lift, add=TRUE,col="green")

Result :

Do you know how to interpret this curve ? Is it exactly the same principle than with population% in abscissa?

Answer 1

The underlying idea of a lift chart is really the same, whether using a population (generally a population response to some marketing effort) or a prediction success rate.

In this case, we're looking at the improvement of predictions as a function of the unpredicted values. For example, at the level where a naive effort could produce a 20% rate of positive prediction, the model you have charted in blue would produce about a 1.23 multiple of that, or approximately 24.6%.

This makes sense visually, as the naive prediction rate trends toward 1, the possible improvement multiple declines sharply as well and in this case the difference between the models contract and necessarily converge at rpp = 1.0

The correct interpretation of this lift chart is that the model plotted in red gives the greatest predictive lift.

That is, at least over the visble range of ~.27 to ~.67.