Interpreting Non-Significant Moderated Regression for Multiple Interactions and a Surprisingly Significant Main Effect

Question

I was wondering if someone could explain this to me.

I've carried out a hierarchical regression, entering 9 independent variables at Step 1 (Block 1), and 8 interaction terms at Step 2 (Block 2). The 8 interaction terms relate to X1 interacting with each of the other Xs. X1 is dummy coded, X2 and onwards are centred.

I'm confused by the results. Model 1 is significant, but Model 2 isn't. From what I understand, this means none of the interactions are significant enough to interpret in Model 2, and so I should focus on Model 1. Please correct me if I'm wrong.

Surprisingly, Model 1 shows 2 out of 9 Xs are significant, when really only 1 should be (i.e. X1, the dummy coded experimental conditions). Purely out of interest, I used a simple regression to check X2 (the surprisingly significant main effect) against my dependent variable, and it came back F(1, 162) = .174, p = .677, with an R2 of .001. The "significant" independent variable in Model 1 isn't significant outside of the Model. Is there any reason why an IV would be significant when grouped with other IVs but not on its own?

Can anyone explain where I go from here, now that Model 2 is non-significant. Theoretically, at least one of the interactions 'could' have moderated the relationship, though the literature's murky on it. I'm considering using Hayes' Process to check each interaction separately, but using Bonferroni to correct for the repeated tests will reduce my alpha considerably.

Answer 1

The concept of significance is quite subtle, particularly when more than one test is run. If you estimate more parameters, this means that under null hypothesis there will be more variation in the estimators. This means that the threshold for significance is higher (because more deviation from null values will happen by random variation even if the H0 is true). For this reason it can occasionally happen that a model with more parameters (model 2 with the interactions) is not significant at the same time when a model that has fewer of the same parameters (your model 1) is significant.

This is admittedly hard to interpret, but I'd suspect that model 1 was only marginally significant and model 2 maybe marginally insignificant. Overall this seems to be a borderline situation with some weak indication that main effects play a role overall, but no conclusive evidence. You need to keep in mind that fixed significance thresholds such as $\alpha=0.05$ are largely arbitrary, and p=0.03 is really not that different from p=0.07, despite one being "significant" at 5%-level whereas the other one is not.

"Surprisingly, Model 1 shows 2 out of 9 Xs are significant, when really only 1 should be (i.e. X1, the dummy coded experimental conditions)." I'm not sure what you mean by "only 1 should be" - how can you be sure that the second one truly is a null effect? Even if it is, testing at 5%-level makes the probability 5% that a true null effect will come out significant, and if you test lots of effects (like, e.g., 9 main effects and 8 interactions), these things happen ultimately with pretty large probability (this is why the Bonferroni correction exists, see below).

" Is there any reason why an IV would be significant when grouped with other IVs but not on its own?" When tested together with other independent variables, the procedure takes dependency among independent variables into account. For example the situation could be that X2 and X3 both have positive influence on Y but X2 is negatively correlated with X3. If you then build a model without X3, you may not see the influence of X2 because it is countered by the influence of X3, which is not in the model. If you put both of them in the model, the test will be adjusted for the correlation. (X3 may still be insignificant, particularly with a not so big sample size, because there is no guarantee that every influential variable will be significant.)

"Can anyone explain where I go from here, now that Model 2 is non-significant." I don't think you can construct a positive statement for any interaction out of this. So you've got to say "no evidence for any interaction existent". Regarding the main effects, as I said, you could say "weak indication".

"I'm considering using Hayes' Process to check each interaction separately, but using Bonferroni to correct for the repeated tests will reduce my alpha considerably." ...and Bonferroni is right about that.