Regression coefficients that flip sign after including other predictors

Question

Imagine

• You run a linear regression with four numeric predictors (IV1, ..., IV4)
• When only IV1 is included as a predictor the standardised beta is +.20
• When you also include IV2 to IV4 the sign of the standardised regression coefficient of IV1 flips to -.25 (i.e., it's become negative).

This gives rise to a few questions:

• With regards to terminology, do you call this a "suppressor effect"?
• What strategies would you use to explain and understand this effect?
• Do you have any examples of such effects in practice and how did you explain and understand these effects?

Answer 1

Multicollinearity is the usual suspect as JoFrhwld mentioned. Basically, if your variables are positively correlated, then the coefficients will be negatively correlated, which can lead to a wrong sign on one of the coefficients.

One check would be to perform a principal components regression or ridge regression. This reduces the dimensionality of the regression space, handling the multicollinearity. You end up with biased estimates but a possibly lower MSE and corrected signs. Whether you go with those particular results or not, it's a good diagnostic check. If you still get sign changes, it may be theoretically interesting.

UPDATE

Following from the comment in John Christie's answer, this might be interesting. Reversal in association (magnitude or direction) are examples of Simpson's Paradox, Lord's Paradox and Suppression Effects. The differences essentially relate to the type of variable. It's more useful to understand the underlying phenomenon rather than think in terms of a particular "paradox" or effect. For a causal perspective, the paper below does a good job of explaining why and I'll quote at length their introduction and conclusion to whet your appetite.

• The role of causal reasoning in understanding Simpson's paradox, Lord's paradox, and the suppression effect: covariate selection in the analysis of observational studies

Tu et al present an analysis of the equivalence of three paradoxes, concluding that all three simply reiterate the unsurprising change in the association of any two variables when a third variable is statistically controlled for. I call this unsurprising because reversal or change in magnitude is common in conditional analysis. To avoid either, we must avoid conditional analysis altogether. What is it about Simpson's and Lord's paradoxes or the suppression effect, beyond their pointing out the obvious, that attracts the intermittent and sometimes alarmist interests seen in the literature?

[...]

In conclusion, it cannot be overemphasized that although Simpson's and related paradoxes reveal the perils of using statistical criteria to guide causal analysis, they hold neither the explanations of the phenomenon they purport to depict nor the pointers on how to avoid them. The explanations and solutions lie in causal reasoning which relies on background knowledge, not statistical criteria. It is high time we stopped treating misinterpreted signs and symptoms ('paradoxes'), and got on with the business of handling the disease ('causality'). We should rightly turn our attention to the perennial problem of covariate selection for causal analysis using non-experimental data.

Answer 2

I believe effects like these are frequently caused by collinearity (see this question). I think the book on multilevel modeling by Gelman and Hill talks about it. The problem is that IV1 is correlated with one or more of the other predictors, and when they are all included in the model, their estimation becomes erratic.

If the coefficient flipping is due to collinearity, then it's not really interesting to report, because it's not due to the relationship between your predictors to the outcome, but really due to the relationship between predictors.

What I've seen suggested to resolve this problem is residualization. First, you fit a model for IV2 ~ IV1, then take the residuals of that model as rIV2. If all of your variables are correlated, you should really residualize all of them. You may choose do to so like this

rIV2 <- resid(IV2 ~ IV1)
rIV3 <- resid(IV3 ~ IV1 + rIV2)
rIV4 <- resid(IV4 ~ IV1 + rIV2 + rIV3)

Now, fit the final model with

DV ~ IV1 + rIV2 + rIV3 + rIV4

Now, the coefficient for rIV2 represents the independent effect of IV2 given its correlation with IV1. I've heard you won't get the same result if you residualized in a different order, and that choosing the residualization order is really a judgment call within your research.

Answer 3

See Simpson's Paradox. In short the main effect observed can reverse when an interaction is added to a model. At the linked page most of the examples are categorical but there is a figure at the top of the page one could imagine continuously. For example, if you have a categorical and continuous predictor then the continuous predictor could easily flip sign if the categorical one is added and within each category the sign is different than for the overall score.