7
$\begingroup$

I have run 3 separate logistic regressions and would somehow like to summarize how well the model fits the data graphically.

Any suggestions?

$\endgroup$
  • $\begingroup$ Not a complete answer, but this blog post over on the Statistics Forum by Michael Lavine may be a good start for some ideas. It is mainly aimed at identifying outliers, so not a complete picture of model fit I suppose. $\endgroup$ – Andy W Aug 25 '11 at 14:49
  • $\begingroup$ What do you mean by "fits"? And what do you mean by "data"? As the logistic model estimates probabilities and the observed Y=0 or 1, these are not easy questions. $\endgroup$ – Frank Harrell Aug 25 '11 at 15:00
  • $\begingroup$ @Frank Harrel I'm trying to see how well a model predicts the observed values which are both binary. I was just trying to ask how I could graphically present the different AIC's or other measures obtained $\endgroup$ – upabove Aug 25 '11 at 15:15
  • $\begingroup$ "Well" is hard to define here. I'd suggest fitting a model that has the right number of parameters for the effective sample size (allowing multiple terms for apriori strong variables not known to act linearly), putting in subject-matter-driven interaction terms, then plotting predicted values from that model. Partial effect plots are very helpful. The R rms package makes this easy. See biostat.mc.vanderbilt.edu/wiki/pub/Main/RmS/rms.pdf $\endgroup$ – Frank Harrell Aug 25 '11 at 15:36
  • $\begingroup$ Why not a simple bar graph that shows the proportion which it succeeds at predicting or fails at predicting, and the proportion of times it gives a false positive / false negative? I think that's something even a layman would understand equally well. $\endgroup$ – Brandon Bertelsen Sep 5 '11 at 22:32
9
$\begingroup$

First I want to reiterate what others have said in the comments: this is in general a difficult problem with no easy solution.

With that in mind, one way of assessing model fit with logistic regression models when your primary interest is simply predictive power is to plot ROCs. Here's an example (with caveats) using the pROC package in R:

library(pROC)

#Fit three logistic models
m1 <- glm(case ~ age,data=infert,family=binomial())
m2 <- glm(case ~ age + parity + education + induced,
          data=infert,family=binomial())
m3 <- glm(case ~ age+parity+education+spontaneous+induced,
          data=infert,family=binomial())

#Plot the three ROCs, adding the AUC for each
plot.roc(infert$case,fitted(m1),print.auc = TRUE)
plot.roc(infert$case,fitted(m2),add=TRUE,col = "blue",
         print.auc=TRUE,print.auc.y = 0.45)
plot.roc(infert$case,fitted(m3),add = TRUE,col = "red",
         print.auc = TRUE,print.auc.y = 0.4)

enter image description here

The basic idea is that more area under the curve indicates a "better" model, from the perspective of prediction only.

Note that my example is meant only to illustrate how to make such a graph, as I've plotted these curves using the in-sample fitted values, rather than those obtained using cross validation, bootstrapping or a hold-out test set, which in general is preferable.

I'll end by repeating my warning that this is far from the way to assess model fit. It is just a way that is limited to assessing predictive accuracy.

$\endgroup$
  • $\begingroup$ what do you mean by the more are under the curve? how is the curve defined? $\endgroup$ – upabove Aug 25 '11 at 17:29
  • $\begingroup$ @Daniel - See here for a basic discussion of ROC curves. In my example, the red curve has more area under it, so it that would indicate a "better" fit. $\endgroup$ – joran Aug 25 '11 at 17:33
  • 2
    $\begingroup$ ROC curves have little to do with model fit. $\endgroup$ – Frank Harrell Aug 25 '11 at 18:36
  • 1
    $\begingroup$ @Daniel - His point, I think, is that this question can't really be answered in a "nutshell". Whole books have been written on this topic. And logistic regression models are a particularly difficult case, at that. I provided an illustration of how to evaluate predictive accuracy, since you mentioned that in particular, but that is only one potential facet of what a statistician would call "model fit". $\endgroup$ – joran Aug 25 '11 at 19:41
  • 1
    $\begingroup$ Some recommendations for assessment of model fit. (1) Make the model fit up-front as much as possible so you don't need to worry so much (e.g., don't assume anything is linear; use regression splines); (2) If you have some hope that the model is simple, add more complex terms to the model and assess their added value (e.g., partial chi-square chunk test); (3) Draw a bootstrap overfitting-corrected smooth calibration curve; (4) Summarize predictive discrimination (show histogram of predicted risks or compute $R^2$ measure or Somers' $D_{xy}$ rank correlation (related to $c$-index or ROC area). $\endgroup$ – Frank Harrell Aug 25 '11 at 23:38
2
$\begingroup$

A good reference for the kind of graphs you seem to want is John Fox, "Effect Displays for Multinomial and Proportional-Odds Logit Models," in Sociological Methodology (2006). See the citations to his earlier work. He implements these techniques in R and S-Plus in his book that accompanies his text on linear regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.