Logistic Regression: classification table vs model results

Question

I am running a binary logistic regression in SPSS. My sample size is 906, but the event of interest is only observed 66 times (so ~7% of the sample). I read several threads on this site and posts from Dr. Frank Harrell about rare event logistic regression, but still have a few questions.

The overall model of my regression is significant and several variables are significant. However, my concern is with the classification table. The overall classification in Block 0 is 94.2% (predicted 0 events occurring, which makes sense due to the low frequency). After running the model, the classification becomes 94.1%, but predicts several events occurring. This is with the cut at 0.5. If I adjust the cut to be 0.2 (which I read on another thread on this site for rare event logistic regressions), the overall classification drops from 94.2% in Block 0 to 91.6% in Block 1. I currently have 7 variables in my model, however this could be reduced if needed.

My overall question is: How do I balance interpreting the overall classification table with the regression results (odds ratio, p-value, etc. of the variables used in the regression)? I am concerned that since the overall classification table did not improve after running the regression that I cannot use the results from the model.

Could the low frequency of the event be influencing the classification table, but the regression results are still usable?

Thank you very much for any guidance!

Answer 1

Welcome to the site.

When your event is rare, you can (as you noted) get quite good overall classification results by just saying "no events happen". But that's not useful. This is why you change the cut. And when you change the cut, you can expect a reduction (or, at least, no improvement) in overall classification. But that's why we also look at specificity and sensitivity and positive predicted value and so on (there are a bunch of these statistics).

One basic decision you have to make is the "cost" of the different types of error. You can make two errors: Say "no event" when it is, or say "event" when it is not. Which is worse? How much worse? This depends on the context. Either one could be much worse, or they could be about equal.

The ORs and associated statistics answer a very different question. They are about how much the independent variables affect things.

If an event happens 1 in 1000 times, but it happens 10 times as often for some group, that's still only 1 in 100 times! Even there, you would be right 99 out of 100 times by saying "nope".

Suppose, just for example, you wanted to predict whether a person was a professional basketball player. Well, there are about 100,000 of them. That's 1 in 80,000. Now you find people who are 2 meters tall and find that they are 50 times more likely (I'm just guessing) to be professional basketball players. That's still only 1 in 1600!

When you have more than one variable in your equation, it gets a little trickier to see this, but the principle is the same.