Logistic regression model

Question

I have a question about binary logistic regression. I have been trying see which IV are the independent predictors of an outcome of a categorical variable ​in a sample of 680 cases, where 30% of cases are having the outcome of interest (total of ~200 cases). No missing variables.

I have about 50 IV. On univariate analysis, 20 of them are statistically significant (p<0.05). However I have 7 IV that have clinical significance (from previous studies), but did not turn significant in the univariate analyses (p>0.25 in all).

If I do logistic regression on all 27 IV at once, I get 11 variables with p <0.05. Nagelkerke R square value is 0.39, Hosmer and Lemeshow Test is 0.64. None of the SE values are (at least to me) obviously high/different from the rest.

Also, I did the automated forward LR (SPSS), which resulted basically with same eleven variables (with the rest of them removed from the model).

My question is, how can I know if the model is overfitted? In theory, I should have put maximum 20 variables to start with, however it is really hard figuring which ones to remove?

How can I know if a model can sustain testing 27 variables at once?

Thank you!

Answer 1

There are whole books written about this. Harrell's resources on Regression Modeling Strategies are a good source. His course notes are particularly helpful.

Chapter 4 of the course notes, on Multivariable Modeling Strategies, deals with ways to handle multiple predictors that go beyond simply including or removing them; sometimes combining related predictors is a better way to go. Chapter 5 covers validation of regression models in general, including use of bootstrap validation to estimate the optimism introduced by overfitting. Other chapters cover logistic regression specifically.

Unless you are wedded to SPSS, Harrell's rms package in R providesvalidate() and calibrate() functions useful for evaluation of a pre-specified model. If you do data-driven predictor selection of the type you are engaged in, to evaluate overfitting you need to do something like optimism-bootstrap evaluation that covers all of your modeling steps, starting from the original data.