I know some well-known measures are $c$ statistic, Kolmogorov-Smirnov $D$ statistic. However, as far as I know, those statistics take into account only of the rank order of the observations, and is invariant under changing the intercept of the logistic regression model (e.g. in oversampling-correction exercise).
In my current application, I need to depend on the accuracy of the logistic regression to predict probability of event. I know only of qualitative way of assessing models for probability prediction ability, namely by plotting "QQ-plot" of the actual vs predicted probability of event:
- Score the validation dataset using the developed model.
- Rank the observations according to the predicted probability and group into $n$ buckets according to their rank of predicted probability. (First 1/n would go to the first bucket, next 1/n would go to the next ...)
- Calculate the average predicted and actual probability of Event for each bucket.
- Create a scatter plot of Predicted vs Actual - one point for each bucket.
I am wondering:
- Is the "Q-Q plot" I mentioned above a legitimate way to assess predictive performance of models developed from logistic regression? If so, where may I find more reference for that?
- Is there any known quantitative way to assess the probability prediction ability of this kind of model?