Should I report the pseudo $R^2$ value for full or final logistic regression model after removing NA's & running stepwise selection?

Question

I'm working with a logistic regression model in r.

model <- glm(response~., family="binomial", data)

and I'm using DescTools::PseudoR2(model, which="Nagelkerke") to get an estimate of model fit.

My dataset has missing values, so if I want to do stepwise selection I have to remove these missing values using na.omit(), but doing so drops 37 out of 187 rows of data.

data2  <- na.omit(data) 
model2 <- glm(response~., family="binomial", data2)

Comparing the two models before using stepwise, I noticed a significant drop in my pseudo $R^2$ value:

DescTools::PseudoR2(model,  which="Nagelkerke")  # 0.6496515  
DescTools::PseudoR2(model2, which="Nagelkerke")  # 0.4652934

After using model 2 to run a backward step based on AIC, my pseudo $R^2$ drops to around .35, which makes sense with fewer variables.

I'm wondering which $R^2$ result I should trust when presenting the model?

Could I present the $R^2$ from model 1 and build a model with the variables kept from the step model, but use the non-NA removed data or is this an inflated and disingenuous estimate?

Answer 1

The short answer to your question is that you cannot use the pseudo $R^2$ from the full model. Your intuition is correct that it is inflated and disingenuous.

That said, you should not use the pseudo $R^2$ from the final model, either. Part of the reason is that pseudo $R^2$ does not really mean what people think it means, and part of the reason is that any attempt to assess goodness of fit after stepwise selection will be irrevocably flawed.

Put simply, you should not use stepwise selection at all.

From there, decide how you want to assess your model. Generally, regression models are designed to pick out conditional means (i.e., the mean of $Y$ when $X$ equals some particular value). So goodness of fit would mean that the model's fitted values are approximately right. You could assess this in various ways, such as plots or tests against saturated models, etc. Alternatively, you could see how well the model predicts out of sample. This could be done via cross validation and Brier scores or calibration.

The question of dropping the NA's is a different issue. That is called 'complete case analysis'. It is valid (but somewhat underpowered) under the assumption that the missingness is MCAR, and often still valid even under MAR (depending on the specifics of the situation).

Here are some threads to read that might help you:

• Algorithms for automatic model selection
• Why are p-values misleading after performing a stepwise selection?
• Sane stepwise regression?
• Backward selection for Cox model using R
• Which pseudo-$R^2$ measure is the one to report for logistic regression (Cox & Snell or Nagelkerke)?
• Is $R^2$ useful or dangerous?
• Is listwise deletion / complete case analysis biased if data are not missing completely at random?