How to compare (probability) predictive ability of models developed from logistic regression?

Question

I know some well-known measures are $c$ statistic, Kolmogorov-Smirnov $D$ statistic. However, as far as I know, those statistics take into account only of the rank order of the observations, and is invariant under changing the intercept of the logistic regression model (e.g. in oversampling-correction exercise).

In my current application, I need to depend on the accuracy of the logistic regression to predict probability of event. I know only of qualitative way of assessing models for probability prediction ability, namely by plotting "QQ-plot" of the actual vs predicted probability of event:

1. Score the validation dataset using the developed model.
2. Rank the observations according to the predicted probability and group into $n$ buckets according to their rank of predicted probability. (First 1/n would go to the first bucket, next 1/n would go to the next ...)
3. Calculate the average predicted and actual probability of Event for each bucket.
4. Create a scatter plot of Predicted vs Actual - one point for each bucket.

I am wondering:

1. Is the "Q-Q plot" I mentioned above a legitimate way to assess predictive performance of models developed from logistic regression? If so, where may I find more reference for that?
2. Is there any known quantitative way to assess the probability prediction ability of this kind of model?

Answer 1

There are many good ways to do it. Here are some examples. These methods are implemented in the R rms package (functions val.prob, calibrate, validate):

1. loess nonparametric full-resolution calibration curve (no binning)
2. Spiegelhalter's test
3. Brier score (a proper accuracy score - quadratic score)
4. Generalized $R^2$ (a proper accuracy score related to deviance)
5. Calibration slope and intercept

For comparing two models with regard to discrimination, the likelihood ratio $\chi^2$ test is the gold standard.

Four of the above approaches, and other approaches, are covered in the 2nd edition of my book Regression Modeling Strategies (coming in 2015-09) and in my course notes that go along with the book, available from https://hbiostat.org/rmsc .

The Brier score can be decomposed into discrimination and calibration components. Along with the Brier score and Spiegelhalter's test, the nonparametric calibration curve can detect errors in the intercept.

Answer 2

The AUROC (which is related to Kolmogorov Smirnov) is not only invariant to a change in coefficient, it is invariant for any order-preserving transformation and consequently it tells how well you predict the ranking of the subjects.

A test for checking whether your probabilities are well predicted is e.g. the Hosmer-Lemeshow test (see e.g. http://media.hsph.edu.vn/sites/default/files/Statistics%20eBook%20-%20Hosmer,%20Lemeshow%20-%20Applied%20Logistic%20Regression.pdf).

There may be other tests, but that depends on your problem. E.g. if you use the logistic regression in the context of predicting company failure and your goal is not to predict probabilities but to predict rating 'classes'.