1
$\begingroup$

I'm running a multiple linear OLS regression (X => Y) on a sample with 125 cases. My regression has two moderator variables (Z and W). Z and W correlate at about .4 but Z and W are two distinct/discriminant concepts (empirically tested - multicolinearity is not present (VIFs are low)).

When including each interaction term individually (XZ alone without XW and vice versa) the interaction terms are significant. However, when I include both interaction terms (XZ and XW) in a full model both become insignificant.

Theoretically speaking, both interaction terms change the relationship between X and Y similarly but through different mechanisms. That is, both interaction terms increase employee satisfaction which strengthens the relationship between X and Y (but Z and W do so through different mechanisms since Z and W are distinct concepts).

How would you explain the insignificant interaction terms in the full model? Is this some kind of suppression effect?

$\endgroup$
  • 1
    $\begingroup$ just wondering if you have run a model with the 3-way interaction...¿does that change the pattern of significances? $\endgroup$ – Gregg H May 25 at 16:02
  • $\begingroup$ Good point. I checked for that already. I do not find a 3-way interaction but its definitely theoretically possible. $\endgroup$ – user18075 May 25 at 16:08
  • $\begingroup$ one more follow-up query: ¿what is the size of the $∆R^2$ for the 2 2-way interaction model compared to the main effects model? $\endgroup$ – Gregg H May 25 at 19:36
  • 1
    $\begingroup$ Also remember, that The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant stat.columbia.edu/~gelman/research/published/signif4.pdf Further, as you add predictors, it is - on average - to be expected that your p-values increase. So be careful to not read too much into the observed pattern - it might be pure noise (especially since your dataset is not very large). $\endgroup$ – Martin Modrák May 29 at 12:16
2
$\begingroup$

I have been struggling with how best to answer this question specific to the context provided, and I'm not sure if my attempt is going to clarify what is happening...or just make things more confusing.

In brief, this is probably an example of a confounding effect. However, the tricky part is that it is confounding via the interaction (e.g., difference in slopes), and I'm not sure you have distinct groups in this context.

That said, here's a context we can use: (1a) we will predict lung capacity from height and gender. If we start by predicting lung capacity from gender, we would find a statistically significant difference. However, if we predict lung capacity from both gender and height, we probably will still see a significant effect by gender, but the gender coefficients between the bivariate model (gender only) and the multiple regression model (gender and height) are probably going to be different. This is because height would be a confounding variable...the two gender groups will have different average heights, and consequently, when we include both in the model, the gender effect in the multiple regression (MR) model is reporting the effect AFTER having accounted for the effect from height.

Now, let me move to a slightly different scenario that is closer to what is probably happening in your data: (1b) this time we will predict lung capacity from treatment group and height. It is possible to run a set of models in which treatment is significant in the bivariate model (treatment effect only), but it is not significant in the MR model (treatment and height effects). Now, the issue is the same as in the example above...but the effect may not be as pronounced. You might think, well, let's just run a t-test to see if the heights are different between the treatment groups. It turns out, that even if there is a difference that doesn't reach significance, it is still possible for that "small" difference to result in a confounding effect. That is, the small difference in heights among the treatment groups may be enough to cause the significance of the treatment predictor to change from the bivariate model to the MR model.

Now, two issues make it tricky to extrapolate to your context. The first is that I was looking at a confounding variable affecting the effect of a grouping variable. It sounds as though all your variables are scalar (not categorical). Well, in this case, you can imagine splitting your data set into the low and high X group. This would allow us to transfer the concept from a categorical variable to a scalar variable. The much trickier part is that you are actually looking at an interaction effect (which is interpreted as the change of slope for the different groups...or in your case, for the differing levels of the X variable).

In your context, accounting for the difference of slopes with only one interaction is coming up significant. However, this affect is confounded by the other interaction term. That is, when both interaction terms are included, the effect appears to disappear.

I'll stop here, and provide any follow-up as requested.

$\endgroup$
  • $\begingroup$ Sorry for the delay. Unfortunately, I had no time to deal with your answer in detail yet. I will catch up on that as soon as possible. $\endgroup$ – user18075 May 29 at 7:54
  • $\begingroup$ Again, sorry for the delay. I really appreciate your long and detailed answer. It helped me a lot in understanding this issue. Thank you very much. However, I had to change my empirical model in the meantime (adding/changing control variables), which resulted in the fact that one of two interaction effects is now significant in the full model. $\endgroup$ – user18075 Jun 5 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.