31
$\begingroup$

Imagine

  • You run a linear regression with four numeric predictors (IV1, ..., IV4)
  • When only IV1 is included as a predictor the standardised beta is +.20
  • When you also include IV2 to IV4 the sign of the standardised regression coefficient of IV1 flips to -.25 (i.e., it's become negative).

This gives rise to a few questions:

  • With regards to terminology, do you call this a "suppressor effect"?
  • What strategies would you use to explain and understand this effect?
  • Do you have any examples of such effects in practice and how did you explain and understand these effects?
$\endgroup$
  • $\begingroup$ How would you explain a situation where coefficients change signs when including predictors but there definately isn't any multicollinearity involved (as low VIF values would suggest)? Interestingly, though, when including predictors, the sign changed to what I initially expected it to be (positive). It was negative in a simple one independant variable regression (correlation matrix showed a minimal negative correlation with the dependant variable) but instantly turned positive with other predictors included. $\endgroup$ – user35283 Nov 25 '13 at 9:31
  • $\begingroup$ @John could you delete your comment and post your question as a separate question on this site (i.e., using "ask question up the top". If you feel that your question is related to this question, then add a link to this question in your new question. $\endgroup$ – Jeromy Anglim Nov 26 '13 at 4:38
  • 2
    $\begingroup$ A paper I wrote with Seth Dutter might help to clarify things. It is written mainly from a geometric perspective. Here is the link: arxiv.org/abs/1503.02722. -Brian Knaeble, B., & Dutter, S. (2015). Reversals of Least-Squares Estimates and Model-Independent Estimation for Directions of Unique Effects. arXiv preprint arXiv:1503.02722. $\endgroup$ – user78992 Jun 5 '15 at 0:44
26
$\begingroup$

Multicollinearity is the usual suspect as JoFrhwld mentioned. Basically, if your variables are positively correlated, then the coefficients will be negatively correlated, which can lead to a wrong sign on one of the coefficients.

One check would be to perform a principal components regression or ridge regression. This reduces the dimensionality of the regression space, handling the multicollinearity. You end up with biased estimates but a possibly lower MSE and corrected signs. Whether you go with those particular results or not, it's a good diagnostic check. If you still get sign changes, it may be theoretically interesting.

UPDATE

Following from the comment in John Christie's answer, this might be interesting. Reversal in association (magnitude or direction) are examples of Simpson's Paradox, Lord's Paradox and Suppression Effects. The differences essentially relate to the type of variable. It's more useful to understand the underlying phenomenon rather than think in terms of a particular "paradox" or effect. For a causal perspective, the paper below does a good job of explaining why and I'll quote at length their introduction and conclusion to whet your appetite.

Tu et al present an analysis of the equivalence of three paradoxes, concluding that all three simply reiterate the unsurprising change in the association of any two variables when a third variable is statistically controlled for. I call this unsurprising because reversal or change in magnitude is common in conditional analysis. To avoid either, we must avoid conditional analysis altogether. What is it about Simpson's and Lord's paradoxes or the suppression effect, beyond their pointing out the obvious, that attracts the intermittent and sometimes alarmist interests seen in the literature?

[...]

In conclusion, it cannot be overemphasized that although Simpson's and related paradoxes reveal the perils of using statistical criteria to guide causal analysis, they hold neither the explanations of the phenomenon they purport to depict nor the pointers on how to avoid them. The explanations and solutions lie in causal reasoning which relies on background knowledge, not statistical criteria. It is high time we stopped treating misinterpreted signs and symptoms ('paradoxes'), and got on with the business of handling the disease ('causality'). We should rightly turn our attention to the perennial problem of covariate selection for causal analysis using non-experimental data.

$\endgroup$
  • 1
    $\begingroup$ Thanks for the suggestion to explore ridge or PCA regression. Just a side point regarding your comment "if your variables are positively correlated, then the coefficients will be negatively correlated leading to sign reversal.": positively correlated predictors do not typically lead to sign-reversal. $\endgroup$ – Jeromy Anglim Aug 12 '10 at 7:11
  • $\begingroup$ Sorry, that's a botched one line explanation written in haste. Fixed now, thanks. $\endgroup$ – ars Aug 12 '10 at 7:19
  • $\begingroup$ Great point about the importance of causal mechanisms. $\endgroup$ – Jeromy Anglim Aug 13 '10 at 2:24
14
$\begingroup$

I believe effects like these are frequently caused by collinearity (see this question). I think the book on multilevel modeling by Gelman and Hill talks about it. The problem is that IV1 is correlated with one or more of the other predictors, and when they are all included in the model, their estimation becomes erratic.

If the coefficient flipping is due to collinearity, then it's not really interesting to report, because it's not due to the relationship between your predictors to the outcome, but really due to the relationship between predictors.

What I've seen suggested to resolve this problem is residualization. First, you fit a model for IV2 ~ IV1, then take the residuals of that model as rIV2. If all of your variables are correlated, you should really residualize all of them. You may choose do to so like this

rIV2 <- resid(IV2 ~ IV1)
rIV3 <- resid(IV3 ~ IV1 + rIV2)
rIV4 <- resid(IV4 ~ IV1 + rIV2 + rIV3)

Now, fit the final model with

DV ~ IV1 + rIV2 + rIV3 + rIV4

Now, the coefficient for rIV2 represents the independent effect of IV2 given its correlation with IV1. I've heard you won't get the same result if you residualized in a different order, and that choosing the residualization order is really a judgment call within your research.

$\endgroup$
  • $\begingroup$ Thanks for the answer. I had these thoughts. (a) Multicollinearity: I agree. Wihtout it, the coefficients should not change. (b) Is it interesting? I actually think that the sign flipping can have interesting theoretical interpretations in some instances; but perhaps not from a pure prediction perspective. (c) Residualisation: I'd be keen to hear what other people think of this approach. $\endgroup$ – Jeromy Anglim Aug 12 '10 at 6:05
  • $\begingroup$ I'm not sure if multicollinearity could be interesting. Say you had some outcome O, and your predictors are Income and Father's Income. The fact that Income is correlated with Father's Income is intrinsically interesting, but that fact would be true no matter the value of O. That is, you could establish that O's predictors are all collinear without ever collecting your outcome data, or even knowing what the outcome is! Those facts shouldn't get especially more interesting once you know that O is really Education. $\endgroup$ – JoFrhwld Aug 12 '10 at 6:25
  • $\begingroup$ I'm suggesting that the suppressor effect can be theoretically interesting, of which presumably multicollinearity provides a starting point for an explanation. $\endgroup$ – Jeromy Anglim Aug 12 '10 at 6:35
5
$\begingroup$

See Simpson's Paradox. In short the main effect observed can reverse when an interaction is added to a model. At the linked page most of the examples are categorical but there is a figure at the top of the page one could imagine continuously. For example, if you have a categorical and continuous predictor then the continuous predictor could easily flip sign if the categorical one is added and within each category the sign is different than for the overall score.

$\endgroup$
  • $\begingroup$ Good point. All the examples of Simpson's Paradox apply to categorical variables. Is the concept of a supressor variable the numeric equivalent? $\endgroup$ – Jeromy Anglim Aug 12 '10 at 7:55

protected by Glen_b -Reinstate Monica Oct 10 '15 at 16:45

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.