Strange behavior of predictive model gain curve

Question

When building predictive models (binary target), one of the principal methods I use for determining how useful the model is, is to plot the true proportion of y=1 values for each decile of the probability score. [These models are marketing models involving rare events and accuracy or the like are not suitable methods for validation].

For example, I build a model on a training set and use the model to score the hold out test set. Then

• Rank the scored data descending by the prediction (probability the target variable is 1)
• Decile (10 equal sized groups) the scored data by this descending prediction (1=highest , 10= lowest)
• Plot the actual proportion of 1's for each decile.

I see from time to time a pattern like the following, which has good ranking discriminatory power but is not strictly monotonically decreasing. Instead, there will be either

1. A hump in the middle OR
2. Misaligned top and bottom of the curve.

Here are fictitious examples of case #1 and case #2, followed by the "ideal".

Question: What causes this type of result - in other words is this a diagnostic for some deficiency in the model that suggest a coarse of corrective action to achieve the ideal?

It seems the first shows the model can separate really good and really poor but not the "mushy middle". The second case is less clear to me. I am not sure a correction for either.

Answer 1

I have seen some of these cases when the variable relation (plot of log odds vs ordered bins of the variable) with the target event is not monotonic. You could plot the log odds of the event to the deciles of each of the variables to check if there is a break in the rank ordering. In case you find a non-monotonic graph, then variable transformations, suitable binning etc. would help establish monotonicity. Hope this helps.

Answer 2

I'm not sure #1 has a specific course of action, other than "add some more predictive features to your model".

There are probably many potential causes to #2, but one can be outliers in the training dataset skewing some of the parameters in a logistic regression model.