# How should strongly correlated covariates for logistic regression be treated?

I have to build a multiple logistic regression model with two strongly correlated covariates (predictor variables). How should they be treated? Am I to exclude one of them from the regression?

There is also a covariate that is a logical (Boolean) variable and is TRUE only for 2 observations (of 107 total). Am I to exclude this covariate from the regression, or is there any sense of taking the covariate into account?

I have 6 covariates and 107 observations. All covariates are Boolean, but may be I will replace some of them with the source numerical variables.

Thank you very much for your answers. I am a mathematician but a novice in applied statistics, so I need detailed answers.

I need to estimate the influence on the outcome of each of the covariates, so I need to calculate odds ratios or something like that.

• How many covariates do you have in total? How much data? How correlated are the 2 in question? – gung - Reinstate Monica Jun 25 '15 at 20:22
• I'd submit that it's impossible for anyone (including the answered already provided below) without gaining a better understanding of the goals of your analysis. For example, if you are simply building a logistic regression model to obtain propensity score estimates and have no need to interpret the coefficients, you don't need to worry about the strongly correlated covariates. They only present problems when you need to interpret them or other covariates with which they may be correlated. – StatsStudent Jun 27 '15 at 6:48

There are at least three regularization strategies to address this multi-collinearity/separation problem.

1) Build a Bayesian regression model that establishes a prior distribution over the regression coefficients that shrinks estimates toward zero, but supplies enough prior probability for the posterior distribution to move toward a signal in the data if it is strong enough. There are several types of priors for this, including but not limited to the Laplace, spike-and-slab, and horseshoe priors. Gelman et al. have a nice paper describing a default prior distribution on coefficients in logistic regression, which pairs well with the bayesglm function they developed in the arm package in R, which allows you to easily build and summarize logistic and other generalized linear models. You can read their paper on the subject here.

2) Penalized regression with L1 norm (LASSO regression), L2 norm (ridge regression), or some combination thereof (the elastic net model). Tibshirani, Hastie, and colleagues have developed a package in R called glmnet, which implements elastic net regression (thus L1 and L2 regression, since they are special cases of the elastic net). This package includes the logit model. There is an excellent vignette of the package, at the end of which you can find useful references on regularization in general as well as for the ridge/LASSO/elastic-net framework. If you want to watch a video version of the vignette, and learn a lot of other stuff, too, I recommend taking their Stanford online course, as well.

3) Another way to deal with multi-collinearity problems in logistic and other generalized linear models is through boosted regression. In boosted regression models, you iteratively aggregate the inferences from many simple models called "base learners". By aggregating the estimates of many simple models, you avoid the curse of dimensionality, and you can also compute variable-importance measures. If you set up your base learners properly, multi-collinearity is no longer an issue. There's a great package in R called mboost, which implements boosted generalized linear models and multilevel generalized linear models. Another reason mboost is great is the variety of base learners available, including non-parametric smoothing splines and random fields. Amazing stuff. Even better is a related package called gamboostLSS, which allows you to build boosted regression models over each of the parameters of our likelihood, not just the mean or some other location parameter.

In your situation, I'd say the best among these methods is either the Gelman et al. recipe or the elastic net option. Of these, I'd prefer the Gelman et al. recipe, because it will yield you not only point estimates, but posterior distributions of the coefficients, as well.

Side note: The beauty of the elastic net and boosting methods, and for some priors the fully Bayesian inference method, is that by regularizing your model, you can also build models with lots of features - even models more features than observations. The regularization procedure in some sense selects those features that are most important while avoiding or at least mitigating the curse of dimensionality.

• In addition to the valuable suggestions in this answer, this Cross Validated page provides a useful entry into extensive discussion about this issue, as relevant to the logistic regression in this question as it is to the linear regression question on the page to which I linked. – EdM Jun 26 '15 at 21:19
• If interpretation of the coefficients is required, then obtaining an index of the variables might also be an effective strategy, for example, by obtaining a principle component scores from the variables. If the correlation is nearly perfect, dropping on the variables is fine too. – StatsStudent Jun 27 '15 at 6:51

You can include weakly correlated variables in your model, but any covariate pairs with r > 0.7 should be reduced to the most relevant predictor.

Covariates with near-zero variance you can exclude from your analysis, since they don't contribute to your model accuracy but increase the degrees-of-freedom.

• This strikes me as advice that may be reasonable in some contexts, but also may not be appropriate in others. I would not use this as a blanket policy. – gung - Reinstate Monica Jun 25 '15 at 21:00
• with all respect, but when a pretty basic question is asked about a logistic regression model with a few variables, then most likely that person is looking for a straight forward answer. But you are right, my answer is pretty sesame-street style in comparison to the other answer given in this thread. – Marijana Vujkovic Jun 26 '15 at 23:14
• Don't take it too hard, it's certainly hard to find the right balance. I don't think there's enough information in the question to customize a really appropriate answer for the OP (& FTR, I didn't downvote you). If I had answered, I probably would have been between your & the other answer's level of detail. – gung - Reinstate Monica Jun 26 '15 at 23:31