# How to compare (probability) predictive ability of models developed from logistic regression?

I know some well-known measures are $$c$$ statistic, Kolmogorov-Smirnov $$D$$ statistic. However, as far as I know, those statistics take into account only of the rank order of the observations, and is invariant under changing the intercept of the logistic regression model (e.g. in oversampling-correction exercise).

In my current application, I need to depend on the accuracy of the logistic regression to predict probability of event. I know only of qualitative way of assessing models for probability prediction ability, namely by plotting "QQ-plot" of the actual vs predicted probability of event:

1. Score the validation dataset using the developed model.
2. Rank the observations according to the predicted probability and group into $$n$$ buckets according to their rank of predicted probability. (First 1/n would go to the first bucket, next 1/n would go to the next ...)
3. Calculate the average predicted and actual probability of Event for each bucket.
4. Create a scatter plot of Predicted vs Actual - one point for each bucket.

I am wondering:

1. Is the "Q-Q plot" I mentioned above a legitimate way to assess predictive performance of models developed from logistic regression? If so, where may I find more reference for that?
2. Is there any known quantitative way to assess the probability prediction ability of this kind of model?

There are many good ways to do it. Here are some examples. These methods are implemented in the R rms package (functions val.prob, calibrate, validate):

1. loess nonparametric full-resolution calibration curve (no binning)
2. Spiegelhalter's test
3. Brier score (a proper accuracy score - quadratic score)
4. Generalized $$R^2$$ (a proper accuracy score related to deviance)
5. Calibration slope and intercept

For comparing two models with regard to discrimination, the likelihood ratio $$\chi^2$$ test is the gold standard.

Four of the above approaches, and other approaches, are covered in the 2nd edition of my book Regression Modeling Strategies (coming in 2015-09) and in my course notes that go along with the book, available from the handouts link at https://biostat.app.vumc.org/wiki/Main/RmS .

The Brier score can be decomposed into discrimination and calibration components. Along with the Brier score and Spiegelhalter's test, the nonparametric calibration curve can detect errors in the intercept.

• Thank you for your quick reply! I have the following follow-up questions: 1) all of them are new to me - is there any resource you would suggest I read to learn how to use them? (i.e. not only get a number, but know what it means) 2) I am using SAS, it provides a Brier score for free when I use SCORE with FITSTAT. Is that the Brier score you are referring to? 3) Just to double-check, all the methods you mention would be able to tell a correct model from a correct model with the wrong intercept, right? (unlike AUROC) Thank you! Aug 1 '15 at 2:55

The AUROC (which is related to Kolmogorov Smirnov) is not only invariant to a change in coefficient, it is invariant for any order-preserving transformation and consequently it tells how well you predict the ranking of the subjects.

A test for checking whether your probabilities are well predicted is e.g. the Hosmer-Lemeshow test (see e.g. http://media.hsph.edu.vn/sites/default/files/Statistics%20eBook%20-%20Hosmer,%20Lemeshow%20-%20Applied%20Logistic%20Regression.pdf).

There may be other tests, but that depends on your problem. E.g. if you use the logistic regression in the context of predicting company failure and your goal is not to predict probabilities but to predict rating 'classes'.

• The Hosmer-Lemeshow test was shown by le Cessie, Hosmer, et al to be arbitrary. It also lacks power. Nonparametric calibration curves have better performance and are less arbitrary. There are also powerful single d.f. tests for model fit: the le Cessie Hosmer sum of squared errors test (related to the Brier score) and the Spiegelhalter test. Aug 1 '15 at 12:24
• Reference for the the le Cessie Hosmer sum of squared errors test (related to the Brier score): Hosmer DW, Hosmer T, Lemeshow S, le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression model. Stat in Med 16:965--980, 1997. Implementation in R: inside-r.org/packages/cran/rms/docs/residuals.lrm Aug 1 '15 at 22:39
• @Frank Harell: do you have some reference on the Spiegelhalter test ?
– user83346
Aug 18 '15 at 4:51
• citeulike.org/user/harrelfe/article/13264888 and it is implemented in the R rms package val.prob function. Aug 18 '15 at 12:11
• @Frank Harell: thanks (+1), maybe you can also help me with this question : stats.stackexchange.com/questions/167483/…?
– user83346
Aug 18 '15 at 12:50