Nonsignificant interaction still causes main effect to flip?

Question

This is my first post so forgive me if this is question has already been asked. I searched around the forum but didn't find anything specific to what I'm looking at currently.

I'll keep things as simple as possible --- (to allay the potential annoyance of semantic sticklers, when I say "the effect is significant" I do mean that it is statistically significantly different from zero, at an alpha level of .05. I'm not making any comment on the substantive meaning or importance of the effects).

Basically I have a two-step linear regression model where the outcome variable is predicted by changes in each of three predictor variables. All of these effects are statistically significant. That's step one.

Then, in customary fashion, I added the interaction of X1 and X2 in the model in step 2. Now I know that adding an interaction term can screw with your main effects, but this still seems unusual to me. In a nutshell:

-- The main effect of X1 is no longer significant and in fact the sign of the beta coefficient flips (negative in model 1, positive in model 2).

-- The main effect of X2 is no longer significant.

-- The main effect of X3 is still significant.

-- aaannnd.... the interaction is NOT significant.

This kind of looks like a case of suppression, but I've never come across an interaction term behaving as a suppressor variable. Moreover, can an interaction term wreak such havoc (yes, I'm catastrophizing) without in fact being statistically significant? I've certainly seen significant interaction effects wipe out main effects, but this case is new to me.

I'm just wondering whether this sort of finding is even worth fussing over, or whether I should just stick with reporting the main effects. Do nonsignificant interaction terms typically cause these sorts of changes for one's main effects?

Any help is appreciated. :)

Answer 1

The main effects in the interaction model are a different kind of thing than the main effects in a model without an interaction. You might think of the main effects when there is no interaction as the average effect of that variable ignoring the other variables. However, once the interaction is included then what you're seeing in main effects is just one of many possible regression coefficients at one particular level of all of the other variables (typically 0). Therefore, you can't interpret main effects in the usual fashion from a model containing interactions among those main effects.

Consider visualization of a two way regression. If you want to assess the interaction you might plot the various slopes of factor A that can occur at various levels of factor B to get some kind of idea of what the interaction looks like. But only one of those slopes is used as the coefficient in the model.

Now you might think that if the effect is strong it will be at least in the same direction no matter what. But consider what I said, it's the coefficient holding the other predictors to 0. What if a predictor's range of values is 300-400? For the interaction that's being held to 0 when calculating other predictors. Even a non-significant interaction can change the coefficient substantially because it's different in kind.

BTW, are you adding all of the second order interactions in as well? Those are important for the model of the 3-way interaction's meaning.