I am running a logistic regression with 5 continuous independent variables (IV). The problem is that IV4 when taken alone has a positive correlation with outcome (coeff > 0), and when taken with the other variables has a negative correlation (coeff < 0). I evaluated correlation between IV4 and the other variables, and the results are: IV4 vs. IV1 (-0.51), IV4 vs. IV2 (-0.48), IV4 vs. IV3 (0.61) and IV4 vs. IV5 (0.73).

I ran other logistic regressions eliminating one at a time all the other variables to look if one of them was responsible for the sign change, and I noticed that when eliminating IV1, the sign of V4 coefficient became positive.

Thus, it seems that IV1 changes the sign of the coefficient of IV4. Is there someone who knows what might be the cause and (possibly) the solution?

Practically, do I have to eliminate the IV4 (or IV1) from the model and explain why?

Thanks a lot for answering

Leonardo Frazzoni, MD

  • 1
    $\begingroup$ Take a look at the correlations of the predictors. Multicollineaity is often responsible for the instability of coefficients. $\endgroup$ – JohnK Jun 13 '15 at 11:40
  • 1
    $\begingroup$ This phenomenon has been extensively discussed here: search for regression sign change. A good one is at stats.stackexchange.com/questions/1580. Add the keyword "logistic" if it's not perfectly clear that the phenomenon is common to all linear regression models. To identify which variables to include in models (it's a huge subject), look at threads related to model-selection. $\endgroup$ – whuber Jun 13 '15 at 13:50

JohnK is correct that collinearity is the leading contender in explaining the wrong sign. Unfortunately, you can't look at zero-order (pairwise) correlations and learn anything about collinearity since the issues are deeper and are to be found in the partial or semi-partial correlations conditional on the model inputs.

There are a plethora of methods available to diagnose the causes of collinearity which range from VIFs (variance inflation factors) and residual analyses to eigenvalue decompositions. These multiple regression diagnostics have become widely available ever since Belsey, Kuh and Wallace's 80s book Regression Diagnostics. These MR diagnostics have their logistic regression analogue in a paper by Daryl Pregibon that dates from the 90s titled simply Logistic Regression Diagnostics. However (and this is the sort of thing that makes the more rigidly orthodox academic-types cringe), a suitable workaround would be to apply the more readily available MR diagnostics to your data preliminary to running your LR.

There are those who object to parameter level diagnostics, arguing that the issue is at the level of the overall model and, unless the standard errors are huge, the resulting model should be the focus: does it make theoretical sense? These papers tend to fall in fields where causal explanations are sought and desired (e.g., political science, see Understanding Interaction Models: Improving Empirical Analyses by Brambor, Clark and Golder, 2006). Bear in mind that, with these prescriptions, they are also standing nearly 30 years of research into regression diagnostics on its head. Moreover, these prescriptions are inconsistent (and/or are lightly documented as in the Brambor, et al, example) with much more rigorously documented conclusions based on earlier research (e.g., see Multiple Regression by Aiken and West, particularly wrt considerations regarding regression model interactions) or are guilty of thoughtlessly repeating erroneous myths about the causes of collinearity (e.g., Brambor, et al, state that collinearity is a "small sample* data problem).

So, do you drop the offending IV1 or IV4? Or, following Brambor, et al, check the overall model properties and, assuming the causal and theoretical appropriateness of the fully specified model, do nothing at all? Since I clearly do not subscribe to Brambor, et al's nostrums, I would elect to making a choice between dropping one of the IVs. For me, the choice has always been which one adds greater absolute relative importance to the model? In your case, with only 2 IVs, this is an easier choice and can be based on standardized metrics such as the t- or chi-squared values in the full model. The IV with the larger absolute value would be retained.

| cite | improve this answer | |
  • 3
    $\begingroup$ Unfortunately, you can't look at zero-order (pairwise) correlations and learn anything about collinearity. This is not true. Although multicollinearity does not only depend on pairwise correlations, these correlations are often telling so examining the correlation matrix is an easy diagnostic most of the time. If one is looking for more sophisticated measures, one can look at the eigenvalues of the correlation matrix. $\endgroup$ – JohnK Jun 13 '15 at 13:07
  • $\begingroup$ @JohnK Sorry, but I beg to differ. Correlations typically refer to bivariate linear or monotonic relationships and are highly dependent on sample size. For instance, in the presence of big data or large n, correlations can be quite small but still show significance. How does one diagnose collinearity in this instance? Similarly, small sample sizes can lead to large correlations in magnitude when the underlying relationships are actually quite independent. My point is that, as a diagnostic, zero-order correlations aren't that helpful. We're going to have to agree to disagree on this $\endgroup$ – Mike Hunter Jun 13 '15 at 14:26
  • $\begingroup$ @JohnK Let me be even more specific about how misleading your suggestion can be. Having been in analytics for more than a couple of years, I am as informed as anybody else regarding the many myths and rules of thumb people have developed over the years. Most of these are legacies from pre-computer era. I've done the head-to-head comparisons between pair-wise correlations and multivariate collinearity diagnostics. The answer does not confirm your assertion that pair-wise correlations tell you something. Any decisions based on them are ill-informed, at best, and in today's world unnecessary $\endgroup$ – Mike Hunter Jun 13 '15 at 15:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.