Timeline for Which pseudo-$R^2$ measure is the one to report for logistic regression (Cox & Snell or Nagelkerke)?
Current License: CC BY-SA 2.5
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 2, 2022 at 12:57 | comment | added | Georg M. Goerg | @whuber Not sure I follow/agree exactly with hosmer and lemeshow: (pseudo) r2 does consider observed and fitted values ( either through residuals or through likelihoods - both consider observed values). I agree that out of sample (business) metric is more important than pseudo r2, but a) pseudo r2 can be computed out of sample too; b) it's a sanity check on whether model does anything useful ( if a (pseudo)r2 out of sample is ~0 - or worse negative - why bother looking at any other metrics downstream? First make sure model is actually predicting anything before jumping to business impact | |
| Sep 10, 2022 at 15:03 | comment | added | whuber♦ | @Shawn H&L don't spend a lot of space on classification error in the 2nd edition. They do devote a lot of discussion in various parts of the book to assessing goodness of fit, which is a related issue. I would expect some expansion of the material on classification in the third edition, but I haven't investigated that. | |
| Sep 9, 2022 at 23:47 | comment | added | Shawn Hemelstrand | Awesome. I will give it a read. Seemed the last book I read spent a lot less time on this topic. | |
| Sep 9, 2022 at 13:07 | comment | added | whuber♦ | @Shawn Yes. I am looking at the 2nd Edition. It has an outstanding discussion of this topic in section 5.2.3, Classification Tables, followed by a brief discussion of the ROC Curve in section 5.2.4. It concludes, "... one cannot compare models on the basis of measures derived from $2\times2$ classification tables since these measures are completely confounded by the distribution of probabilities in the samples upon which they are based." | |
| Sep 9, 2022 at 7:34 | comment | added | Shawn Hemelstrand | @whuber thank you for this useful answer. I am looking at a lot of diagnostics of my logistic regression at the moment. Is classification error rate covered in the Hosmer & Lemeshow book you mentioned? | |
| Oct 25, 2016 at 14:16 | comment | added | Wayne | +1. Also, to expand on a subtle part of your answer, you mention classification error rates, which is plural and should not be confused with accuracy. There are many different kinds of calculations that can come out of a confusion matrix -- accuracy, false positive rate, precision, etc -- and which one we care about depends on the application. Also, you make the distinction of out-of-sample, which is distinct from cross validation, but sometimes confused with it. | |
| Nov 19, 2011 at 14:39 | comment | added | rolando2 | I see. Upon crossvalidation. Good point, thx. | |
| Nov 18, 2011 at 15:16 | comment | added | whuber♦ | @rolando2 Yes, I have. This raises the question of how much the pseudo-$R^2$ ought to go up to justify inclusion of variables. I suspect your "correct classification rate" may refer to the in-sample rate, which of course is biased. If that's correct, then what you read merely compares two inferior statistics. The out of sample rate is far more useful an indicator than the pseudo-$R^2$. | |
| Nov 18, 2011 at 0:49 | comment | added | rolando2 | @whuber: I also tend to gravitate toward correct classif. rates, but I have seen numerous references in textbooks and websites cautioning analysts not to trust them and stressing that pseudo-rsq, despite its limitations, is a fairer metric. I often read something that seems borne out to some degree in my own analyses: that with the addition of a given predictor pseudo-rsq might go up (and other metrics will indicate a benefit from the addition) while correct classification rate fails to, and that one shouldn't trust the latter. Have you given this any thought? | |
| Jul 28, 2011 at 12:31 | vote | accept | Henrik | ||
| Nov 23, 2010 at 21:37 | comment | added | Brandon Bertelsen | thanks for this, helpful for a project I'm working on currently as well - and totally makes sense. | |
| Oct 14, 2010 at 7:21 | vote | accept | Henrik | ||
| Jul 28, 2011 at 12:30 | |||||
| Oct 13, 2010 at 18:31 | comment | added | chl | (+1) I was initially thinking of expanding my response (that came just after yours), but definitely your answer is self-sufficient. | |
| Oct 13, 2010 at 17:46 | history | answered | whuber♦ | CC BY-SA 2.5 |