# Presenting logistic model fit graphically

I have run 3 separate logistic regressions and would somehow like to summarize how well the model fits the data graphically.

Any suggestions?

-
Not a complete answer, but this blog post over on the Statistics Forum by Michael Lavine may be a good start for some ideas. It is mainly aimed at identifying outliers, so not a complete picture of model fit I suppose. –  Andy W Aug 25 '11 at 14:49
What do you mean by "fits"? And what do you mean by "data"? As the logistic model estimates probabilities and the observed Y=0 or 1, these are not easy questions. –  Frank Harrell Aug 25 '11 at 15:00
@Frank Harrel I'm trying to see how well a model predicts the observed values which are both binary. I was just trying to ask how I could graphically present the different AIC's or other measures obtained –  Dbr Aug 25 '11 at 15:15
"Well" is hard to define here. I'd suggest fitting a model that has the right number of parameters for the effective sample size (allowing multiple terms for apriori strong variables not known to act linearly), putting in subject-matter-driven interaction terms, then plotting predicted values from that model. Partial effect plots are very helpful. The R rms package makes this easy. See biostat.mc.vanderbilt.edu/wiki/pub/Main/RmS/rms.pdf –  Frank Harrell Aug 25 '11 at 15:36
Why not a simple bar graph that shows the proportion which it succeeds at predicting or fails at predicting, and the proportion of times it gives a false positive / false negative? I think that's something even a layman would understand equally well. –  Brandon Bertelsen Sep 5 '11 at 22:32

First I want to reiterate what others have said in the comments: this is in general a difficult problem with no easy solution.

With that in mind, one way of assessing model fit with logistic regression models when your primary interest is simply predictive power is to plot ROCs. Here's an example (with caveats) using the pROC package in R:

library(pROC)

#Fit three logistic models
m1 <- glm(case ~ age,data=infert,family=binomial())
m2 <- glm(case ~ age + parity + education + induced,
data=infert,family=binomial())
m3 <- glm(case ~ age+parity+education+spontaneous+induced,
data=infert,family=binomial())

#Plot the three ROCs, adding the AUC for each
plot.roc(infert$case,fitted(m1),print.auc = TRUE) plot.roc(infert$case,fitted(m2),add=TRUE,col = "blue",
print.auc=TRUE,print.auc.y = 0.45)
print.auc = TRUE,print.auc.y = 0.4)


The basic idea is that more area under the curve indicates a "better" model, from the perspective of prediction only.

Note that my example is meant only to illustrate how to make such a graph, as I've plotted these curves using the in-sample fitted values, rather than those obtained using cross validation, bootstrapping or a hold-out test set, which in general is preferable.

I'll end by repeating my warning that this is far from the way to assess model fit. It is just a way that is limited to assessing predictive accuracy.

-
what do you mean by the more are under the curve? how is the curve defined? –  Dbr Aug 25 '11 at 17:29
@Daniel - See here for a basic discussion of ROC curves. In my example, the red curve has more area under it, so it that would indicate a "better" fit. –  joran Aug 25 '11 at 17:33
so basically the highr the value of AUC the better the model? –  Dbr Aug 25 '11 at 17:34
@Daniel - Yes, but please mind all my caveats if you actually use this! There are no shortcuts to assessing model fit! –  joran Aug 25 '11 at 17:38
ROC curves have little to do with model fit. –  Frank Harrell Aug 25 '11 at 18:36

A good reference for the kind of graphs you seem to want is John Fox, "Effect Displays for Multinomial and Proportional-Odds Logit Models," in Sociological Methodology (2006). See the citations to his earlier work. He implements these techniques in R and S-Plus in his book that accompanies his text on linear regression.

-
+1 for citing a Canadian. :) –  Brandon Bertelsen Sep 5 '11 at 21:33