When building predictive models (binary target), one of the principal methods I use for determining how useful the model is, is to plot the true proportion of y=1 values for each decile of the probability score. [These models are marketing models involving rare events and accuracy or the like are not suitable methods for validation].
For example, I build a model on a training set and use the model to score the hold out test set. Then
- Rank the scored data descending by the prediction (probability the target variable is 1)
- Decile (10 equal sized groups) the scored data by this descending prediction (1=highest , 10= lowest)
- Plot the actual proportion of 1's for each decile.
I see from time to time a pattern like the following, which has good ranking discriminatory power but is not strictly monotonically decreasing. Instead, there will be either
- A hump in the middle OR
- Misaligned top and bottom of the curve.
Here are fictitious examples of case #1 and case #2, followed by the "ideal".
Question: What causes this type of result - in other words is this a diagnostic for some deficiency in the model that suggest a coarse of corrective action to achieve the ideal?
It seems the first shows the model can separate really good and really poor but not the "mushy middle". The second case is less clear to me. I am not sure a correction for either.