Presenting logistic model fit graphically

Question

I have run 3 separate logistic regressions and would somehow like to summarize how well the model fits the data graphically.

Any suggestions?

Answer 1

First I want to reiterate what others have said in the comments: this is in general a difficult problem with no easy solution.

With that in mind, one way of assessing model fit with logistic regression models when your primary interest is simply predictive power is to plot ROCs. Here's an example (with caveats) using the pROC package in R:

library(pROC)

#Fit three logistic models
m1 <- glm(case ~ age,data=infert,family=binomial())
m2 <- glm(case ~ age + parity + education + induced,
          data=infert,family=binomial())
m3 <- glm(case ~ age+parity+education+spontaneous+induced,
          data=infert,family=binomial())

#Plot the three ROCs, adding the AUC for each
plot.roc(infert$case,fitted(m1),print.auc = TRUE)
plot.roc(infert$case,fitted(m2),add=TRUE,col = "blue",
         print.auc=TRUE,print.auc.y = 0.45)
plot.roc(infert$case,fitted(m3),add = TRUE,col = "red",
         print.auc = TRUE,print.auc.y = 0.4)

The basic idea is that more area under the curve indicates a "better" model, from the perspective of prediction only.

Note that my example is meant only to illustrate how to make such a graph, as I've plotted these curves using the in-sample fitted values, rather than those obtained using cross validation, bootstrapping or a hold-out test set, which in general is preferable.

I'll end by repeating my warning that this is far from the way to assess model fit. It is just a way that is limited to assessing predictive accuracy.

Answer 2

A good reference for the kind of graphs you seem to want is John Fox, "Effect Displays for Multinomial and Proportional-Odds Logit Models," in Sociological Methodology (2006). See the citations to his earlier work. He implements these techniques in R and S-Plus in his book that accompanies his text on linear regression.