Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a SPSS Output for a logistic regression. This output reports two measure for the model fit, Cox & Snell and Nagelkerke.

So as a rule of thumb, which of these R² measures would you report as the model fit?

Or, which of these fit indices is the one that is usually reported in journals?


Some Background: The regression tries to predict the presence or absence of some bird (capercaillie) from some environmental variables (e.g., steepness, vegetation cover, ...). Unfortunately, the bird did not appear very often (35 hits to 468 misses) so the regression performs rather poorly. Cox & Snell is .09, Nagelkerke, .23.
The subject is environmental sciences or ecology.

share|improve this question
2  
Srikant's answer is certainly a good one (and I have nothing to contribute), but you might want to wait a bit longer than 24 minutes to mark a best answer to see if anybody else has any ideas. –  Matt Parker Oct 13 '10 at 17:05
2  
The excellent UCLA stats help site has an excellent page explaining the various pseudo-$R^2$'s & how they are related to each other. –  gung Mar 14 '13 at 3:47

8 Answers 8

up vote 45 down vote accepted

Normally I wouldn't report $R^2$ at all. Hosmer and Lemeshow, in their textbook Applied Logistic Regression (2nd Ed.), explain why:

In general, [$R^2$ measures] are based on various comparisons of the predicted values from the fitted model to those from [the base model], the no data or intercept only model and, as a result, do not assess goodness-of-fit. We think that a true measure of fit is one based strictly on a comparison of observed to predicted values from the fitted model.

[At p. 164.]

Concerning various ML versions of $R^2$, the "pseudo $R^2$" stat, they mention that it is not "recommended for routine use, as it is not as intuitively easy to explain," but they feel obliged to describe it because various software packages report it.

They conclude this discussion by writing,

...low $R^2$ values in logistic regression are the norm and this presents a problem when reporting their values to an audience accustomed to seeing linear regression values. ... Thus [arguing by reference to running examples in the text] we do not recommend routine publishing of $R^2$ values with results from fitted logistic models. However, they may be helpful in the model building state as a statistic to evaluate competing models.

[At p. 167.]

My experience with some large logistic models (100k to 300k records, 100 - 300 explanatory variables) has been exactly as H & L describe. I could achieve relatively high $R^2$ with my data, up to about 0.40. These corresponded to classification error rates between 3% and 15% (false negatives and false positives, balanced, as confirmed using 50% hold-out datasets). As H & L hinted, I had to spend a lot of time disabusing the client (a sophisticated consultant himself, who was familiar with $R^2$) concerning $R^2$ and getting him to focus on what mattered in the analysis (the classification error rates). I can warmly recommend describing the results of your analysis without reference to $R^2$, which is more likely to mislead than not.

share|improve this answer
    
(+1) I was initially thinking of expanding my response (that came just after yours), but definitely your answer is self-sufficient. –  chl Oct 13 '10 at 18:31
    
@chl I wondered why your response wasn't as encyclopedic as usual :-). –  whuber Oct 13 '10 at 19:45
    
thanks for this, helpful for a project I'm working on currently as well - and totally makes sense. –  Brandon Bertelsen Nov 23 '10 at 21:37
    
@whuber: I also tend to gravitate toward correct classif. rates, but I have seen numerous references in textbooks and websites cautioning analysts not to trust them and stressing that pseudo-rsq, despite its limitations, is a fairer metric. I often read something that seems borne out to some degree in my own analyses: that with the addition of a given predictor pseudo-rsq might go up (and other metrics will indicate a benefit from the addition) while correct classification rate fails to, and that one shouldn't trust the latter. Have you given this any thought? –  rolando2 Nov 18 '11 at 0:49
2  
@rolando2 Yes, I have. This raises the question of how much the pseudo-$R^2$ ought to go up to justify inclusion of variables. I suspect your "correct classification rate" may refer to the in-sample rate, which of course is biased. If that's correct, then what you read merely compares two inferior statistics. The out of sample rate is far more useful an indicator than the pseudo-$R^2$. –  whuber Nov 18 '11 at 15:16

Both indices are measures of strength of association (i.e. whether any predictor is associated with the outcome, as for an LR test), and can be used to quantify predictive ability or model performance. A single predictor may have a significant effect on the outcome but it might not necessarily be so useful for predicting individual response, hence the need to assess model performance as a whole (wrt. the null model). The Nagelkerke $R^2$ is useful because it has a maximum value of 1.0, as Srikant said. This is just a normalized version of the $R^2$ computed from the likelihood ratio, $R^2_{\text{LR}}=1-\exp(-\text{LR}/n)$, which has connection with the Wald statistic for overall association, as originally proposed by Cox and Snell. Other indices of predictive ability are Brier score, the C index (concordance probability or ROC area), or Somers' D, the latter two providing a better measure of predictive discrimination.

The only assumptions made in logistic regression are that of linearity and additivity (+ independence). Although many global goodness-of-fit tests (like the Hosmer & Lemeshow $\chi^2$ test, but see my comment to @onestop) have been proposed, they generally lack power. For assessing model fit, it is better to rely on visual criteria (stratified estimates, nonparametric smoothing) that help to spot local or global departure between predicted and observed outcomes (e.g. non-linearity or interaction), and this is largely detailed in Harrell's RMS handout. On a related subject (calibration tests), Steyerberg (Clinical Prediction Models, 2009) points to the same approach for assessing the agreement between observed outcomes and predicted probabilities:

Calibration is related to goodness-of-fit, which relates to the ability of a model to fit a given set of data. Typically, there is no single goodness-of-fit test that has good power against all kinds of lack of fit of a prediction model. Examples of lack of fit are missed non-linearities, interactions, or an inappropriate link function between the linear predictor and the outcome. Goodness-of-fit can be tested with a $\chi^2$ statistic. (p. 274)

He also suggests to rely on the absolute difference between smoothed observed outcomes and predicted probabilities either visually, or with the so-called Harrell's E statistic.

More details can be found in Harrell's book, Regression Modeling Strategies (pp. 203-205, 230-244, 247-249). For a more recent discussion, see also

Steyerberg, EW, Vickers, AJ, Cook, NR, Gerds, T, Gonen, M, Obuchowski, N, Pencina, MJ, and Kattan, MW (2010). Assessing the Performance of Prediction Models, A Framework for Traditional and Novel Measures. Epidemiology, 21(1), 128-138.

share|improve this answer
    
could you elaborate on the distinction between "goodness of fit" and strength of association or predictive ability? –  Andy W Oct 14 '10 at 13:12
    
@Andy Thanks for pointing that. I realize afterwards that my first sentence does not sound well indeed. I'll update my answer, pls let me know if this ok with you. –  chl Oct 14 '10 at 14:00
    
Thanks for the update and it does clarify the distinction. –  Andy W Oct 15 '10 at 13:24

I found Tue Tjur's short paper on various proposals for a coefficient of determination in logistic models quite enlightening. He does a good job highlighting pros and cons - and of course offers a new definition. Very much recommended (though I have no favorite myself).

share|improve this answer
1  
Thanks for pointing out that paper; somehow I missed it (and it appeared when I was in the middle of a big logistic regression project!). –  whuber Oct 13 '10 at 20:44
1  
For the record, this new definition is $D=\bar{\hat\pi}_1 - \bar{\hat\pi}_0$, which is the mean predicted value for the $1$ responses minus the mean predicted value for the $0$ responses. It can range from $0$ to $1$. Tjur does not dismiss the Nagelkerke pseudo $R^2$, but suggests it lacks the "intuitive appeal" enjoyed by $D$. –  whuber Jul 2 at 15:13

I would have thought the main problem with any kind of $R^2$ measure for logistic regression is that you are dealing with a model which has a known noise value. This is unlike standard linear regression, where the noise level is usually treated as unknown. For we can write a glm probability density function as:

$$f(y_i|\mu_i,\phi)=\exp\left(\frac{y_ib(\mu_i)-c(\mu_i)}{\phi}+d(y_i,\phi)\right)$$

Where $b(.),c(.),d(.;.)$ are known functions, and $\mu_i=g^{-1}(x_i^T\beta)$ for inverse link function $g^{-1}(.)$. If we define the usual glm deviance residuals as

$$d_i^2=2\phi\left(log[f(y_i|\mu_i=y_i,\phi)]-log[f(y_i|\mu_i=\hat{\mu}_i,\phi)]\right)$$ $$=2\phi \left[y_ib(y_i)-y_ib(\hat{\mu}_i)-c(y_i)+c(\hat{\mu}_i)\right]$$

The we have (via likelihood ratio chi-square, $\chi^2=\frac{1}{\phi}\sum_{i=1}^{N}d_i^2$)

$$E\left(\sum_{i=1}^{N}d_i^2\right)=E(\phi\chi^2)\approx (N-p)\phi$$

Where $p$ is the dimension of $\beta$. For logistic regression we have $\phi=1$, which is known. So we can use this to decide on a definite level of residual that is "acceptable" or "reasonable". This usually cannot be done for OLS regression (unless you have prior information about the noise). Namely, we expect each deviance residual to be about $1$. Too many $d_i^2>>1$ and it is likely that an important effects are missing from the model (under-fitting); too many $d_i^2<<1$ and it is likely that there are redundant or spurious effects in the model (over-fitting). (these could also mean model mispecification).

Now this means that the problem for the pseudo-$R^2$ is that it fails to take into account that the level of binomial variation is predictable (provided the binomial error structure isn't being questioned). Thus even though Nagelkerke ranges from $0$ to $1$, it is still not scaled properly. Additionally, I can't see why these are called pseudo $R^2$ if they aren't equal to the usual $R^2$ when you fit a "GLM" with an identity link and normal error. For example, the equivalent cox-snell R-squared for normal error (using REML estimate of variance) is given by:

$$R^2_{CS}=1-\exp\left(-\frac{N-p}{N}\frac{R^2_{OLS}}{1-R^2_{OLS}}\right)$$

Which certainly looks strange.

I think the better "Goodness of Fit" measure is the sum of the deviance residuals, $\chi^2$. This is mainly because we have a target to aim for.

share|improve this answer
    
+1 Nice exposition of the issues hinted at in the comments following Srikant's answer. –  whuber Nov 16 '11 at 14:49

I was also going to say 'neither of them', so i've upvoted whuber's answer.

As well as criticising R^2, Hosmer & Lemeshow did propose an alternative measure of goodness-of-fit for logistic regression that is sometimes useful. This is based on dividing the data into (say) 10 groups of equal size (or as near as possible) by ordering on the predicted probability (or equivalently, the linear predictor) then comparing the observed to expected number of positive responses in each group and performing a chi-squared test. This 'Hosmer-Lemeshow goodness-of-fit test' is implemented in most statistical software packages.

share|improve this answer
3  
The original HL $\chi^2$ GoF test is not very powerful for it depends on categorizing the continuous predictor scale into an arbitrary number of groups; H & L proposed to consider decile, but obviously it depends on the sample size, and under some circumstances (e.g. IRT models) you often have very few people at one or both end of the scale such that cutoffs are unevenly spaced. See A comparison of goodness-of-fit tests for the logistic regression model, Stat. Med. 1997 16(9):965, j.mp/aV2W6I –  chl Oct 14 '10 at 6:36
    
Thanks chi, that's a useful ref, though your j.mp link took me to a BiblioInserm login prompt. Here's a doi-based link: dx.doi.org/10.1002/… –  onestop Oct 14 '10 at 7:20
    
Sorry for the incorrect link... I seem to remember Frank Harrell's Design package features the alternative H&L 1 df test. –  chl Oct 14 '10 at 7:31

I would prefer the Nagelkerke as this model fit attains 1 when the model fits perfectly giving the reader a sense of how far your model is from perfect fit. The Cox & Shell does not attain 1 for perfect model fit and hence interpreting a value of 0.09 is a bit harder. See this url for further info on Pseudo RSquared for an explanation of various types of fits.

share|improve this answer
6  
A "perfect fit" is so far from being attainable in any realistic logistic regression that it seems unfair to use it as a reference or a standard. –  whuber Oct 13 '10 at 17:47
1  
@whuber True but you could use the standard to compare the relative performance of two competing models. Your points of low R^2 in your answer and its implications are good points but if you have (e.g., reviewers demand it etc) to use some form of R^2 then Nagelkerke is preferable. –  user28 Oct 13 '10 at 18:32
1  
@Skridant Yes, still the problem of reviewers that want to see $R^2$ and Bonferroni correction everywhere... –  chl Oct 13 '10 at 19:32
    
@Srikant, @chl: A cynical reading of this thread would suggest just picking the largest R^2 among all those the software reports ;-). –  whuber Oct 13 '10 at 19:44
2  
@chl Offering push-back to reviewers/clients is of course necessary but sometimes we have to be pragmatic as well. If readers do not mis-interpret low R^2 as lack of adequate model performance then the issues raised by @whuber will be mitigated to some extent. –  user28 Oct 13 '10 at 19:46

Despite the arguments against using pseudo-r-squareds, some people will for various reasons want to continue using them at least at certain times. What I have internalized from my readings (and I'm sorry I cannot provide citations at the moment) is that

  • if both C&S and Nag. are below .5, C&S will be a better gauge;
    if they're both above .5, Nag. will; and
    if they straddle .5, punt.

Also, a formula whose results often fall between these two, mentioned by Scott Menard in Applied Logistic Regression Analysis (Sage), is

[-2LL0 - (-2LL1)]/-2LL0.

This is denoted as "L" in the chart below.

enter image description here

share|improve this answer
    
What does this picture show (what does the horizontal axis stand for)? Also, how does the last formula (which looks like a scaled likelihood ratio statistic) differ from Nagelkerke $R^2$ exactly? –  chl Nov 17 '11 at 11:29
    
Analysis #: I tried various analyses with different datasets. Don't have the Nagelkerke formula handy but I bet it's readily available. –  rolando2 Nov 18 '11 at 0:35
    
Paul Allison covers the Nagelkerke formula, which is an upward-adjusted Cox & Snell formula, at statisticalhorizons.com/2013/02. After reading that blog, and generally in the 2-3 years since most of this discussion took place, I've become more convinced that Cox & Snell's underestimates explained variance and that I'm better off averaging C & S and the Nagelkerke result. –  rolando2 Oct 23 '13 at 15:25

Here are two links that discuss an exact non-parametric algorithm which maximizes the accuracy of logistic regression models. If you use this method with your data, it will increase the classification performance of your logistic regression model when applied to the sample.

Example 1: http://onlinelibrary.wiley.com/doi/10.1111/j.1540-5915.1991.tb01912.x/abstract

Example 2: http://epm.sagepub.com/content/54/1/73.abstract

share|improve this answer

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.