Logistic Regression and Omitted Variable Bias

Question

I just want to confirm that I am understanding this correctly. So if logistic regression models have omitted variable bias, does that mean that I should discard any logistic regression models that does not include every significant main effects and their every significant interactions, even if I need to exclude some of the potentially significant interaction terms to avoid multicollinearity or to make an MCMC-GLM algorithm converge?

In my situation: 1. I am fitting a logistic regression model for my binary response against three predictors. When I run glm(y~(A+B+C)^3) in R it does not seem to have any problem with this. However, when I try to run MCMClogit(y~(A+B+C)^3) for a Bayesian power analysis on logistic regression model, R tells me that the algorithm does not converge. The MCMClogit algorithm cannot handle the model involving full interaction, meaning the Bayesian method is subject to omitted variable bias. 2. Because I was being so cautious about the omitted variable bias in logistic regression, I added bunch of interaction terms into my logistic regression model that deem to have significant effect on the response. This causes multicollinearity, which makes the p-values to become unreliable.

Does this mean that I should abandon this Bayesian approach of conducting power analysis? and does omitted variable bias mean that I should include every significant interaction even if it causes multicollinearity?

Answer 1

In practice this issue with omitted-variable bias in logistic regression might not be that much different from what is faced in ordinary least squares (OLS). The added problem in logistic regression is that, unlike OLS, omitting predictors associated with outcome but uncorrelated with the included predictors leads to bias in the coefficient estimates for the included predictors.

A major point of this question is how this principle should inform the inclusion of interaction terms in a logistic model. Although it can be possible to transform 2 variables to have their interaction or product uncorrelated with either of them, that isn't always done in practice and doesn't seem to have been done for the set of all interactions up to 3rd order in the example in this question.* So if interaction terms are correlated with included predictors there can still be an omitted-variable bias when interaction terms are omitted from OLS. To that extent, the issues with interaction terms and omitted-variable bias aren't necessarily different between OLS and logistic regression.

Furthermore, the bias in logistic regression tends to be in a conservative direction: omitting predictors associated with outcome but uncorrelated with the included predictors leads to bias of coefficient estimates toward 0. Depending on the purpose of the modeling, that might be an acceptable tradeoff.

Multicollinearity doesn't necessarily lead to an unreliable model. Yes, there can be large standard errors of individual coefficient estimates. But as this answer puts it:

Finally, consider the actual impact of multicollinearity. It doesn't change the predictive power of the model (at least, on the training data) but it does screw with our coefficient estimates. In most ML applications, we don't care about coefficients themselves, just the loss of our model predictions...

Predictions take into account both the coefficient estimates (including the interaction coefficients) and the coefficient covariance matrix, providing a precision in predictions superior to what one might expect from just looking at the standard error of an individual coefficient.

I have no experience with the Bayesian approach mentioned in this question, and can't say why it didn't converge. But with respect to including interaction terms in logistic regression, the principles to apply in practice aren't necessarily much different from those for OLS. Domain knowledge should be a primary guide. Then apply the standard arts of statistical analysis, choosing a model complexity appropriate to the available data.

---

*It's not clear to me whether predictors can always be transformed to make all their 3-way interactions uncorrelated with the individual predictors.