IV's direct effect changes sign with different interaction terms

Question

I am running a couple of multilevel models. I have a puzzling problem that I am unable to interpret. My non significant IV changes to a Positive (and significant) when I add the first interaction term; this then changes to negative (and significant) when I add another interaction term.

** by IV - I mean independent variable :) sorry about the confusion!**

Here are my models M1 = controls + IV1 +IV2 +IV3 (no significant effect) M2 = controls + IV1 +IV2 + IV3 + IV1*IV2 (here IV1 is positive, IV2 is non significant and interaction is negative) M3 = controls + IV1 +IV2 + IV3 + IV1*IV3 (Here IV1 is negative(!), IV3 is negative, and new interaction term is Positive)

I am not sure what it means when my IV1 term keeps changing sign!!

I am happy to give additional information! Thanks in advance for the help :)

Answer 1

Introducing instrumental variables is likely to cause multicollinearity. (You can check how big this is by calculating the Variance Inflation Factor (VIF); if you use R the package to use is "car".) And multicollinearity often creates exactly the effect you described, namely erratically changing signs of coefficient involved in multicollinearity. Please also note, as you introduce interactions, say IV1*IV2, your original effect of IV1 gets split into IV1 and IV1*IV2. Let $\beta$ be the coefficient for IV1, $\rho$ the coefficient for the interaction IV1*IV2 and mean(IV2) the average value for IV2 in the sample. If you want to calculate what effect a one unit increase of IV1 has in the case of interaction, you need to compute $\beta*1+\rho*1*$mean(IV2). This is called the marginal effect if you want to google it further. Does this make sense?

Answer 2

You've hit on an issue in regression modeling (that include interactions, hierarchical and not) about which there is much disagreement in the literature with many conflicting rules of thumb for its solution. My views are based on Aiken and West's book Multiple Regression which, IMO, has one of the clearest and most thorough expositions on this topic.

Briefly, adding an interaction changes the interpretation of the model. Main effects models are evaluated at the conditional mean of the predictors, whereas models with an interaction are evaluated at zero.

Given that, A&W recommend normalizing your features to a mean of zero before taking an interaction term. This minimizes what some call nonessential dependence -- collinearity -- but doesn't remove all of the dependence that may still be in the data. They also recommend including any predictors (as main effects) used for the interaction in the final model, as opposed to leaving them out of they are not significant. Once an interaction is introduced, the importance of these predictors (main effects) used in the interaction is superceded or overridden by the coefficient and sign of the interaction term. In other words, you should no longer be interested in or try to interpret them (i.e., the main effects used in the interaction).

In the case of HLMs, you would probably want to evaluate the extent to which adding an interaction term(s) has reduced model error in deciding whether to use it or not.

A good explanation of all of this can be found in Judith Singer's book Applied Longitudinal Analysis or in her short, ungated paper, Using SAS PROC MIXED to Fit Multilevel Models, Hierarchical Models, and Individual Growth Models. Forget the SAS part, it's simply a very useful introduction and overview that includes discussion of dealing with interactions in HLMs. Her paper can be found here ... https://www.ida.liu.se/~732G34/info/singer.pdf