I try to get a sense of the correct adjusted Holm-Bonferroni significance levels $\alpha$ when dealing with multiple interaction terms. To simplify, assume one has the following model: $$ y=\beta_0 + \beta_1x_1 + \beta_2x_2+\beta_3x_3+\beta_4x_1x_3+\beta_5x_2x_3 + \varepsilon $$

I would publish any of the following statistical findings (Theories T):

T1a: $\beta_1\ne0$ and $\beta_4=0$

T1b: $\beta_1\ne0$ and $\beta_4\ne0$

T2a: $\beta_2\ne0$ and $\beta_5=0$

T2b: $\beta_2\ne0$ and $\beta_5\ne0$

The crucial point is that I would not publish $\beta_1=0$ and $\beta_4\ne0$ and would also not publish $\beta_2=0$ and $\beta_5\ne0$. Thus, I test $\beta_4$ and $\beta_5$ only conditional on $\beta_1$ and $\beta_2$, respectively, being significant.

Let's assume now, we have the following p-values: 0.01 for $\beta_1$, 0.02 for $\beta_2$, 0.04 for $\beta_4$ and 0.05 for $\beta_5$. We test for the overall significance level $\alpha=0.05$.

Using the Holm-Bonferroni approach, I would assume that the significance level for $\beta_1$ is $\alpha/2=0.25$. That is because the interaction terms $\beta_4$ and $\beta_5$ are never tested before $\beta_1$ and $\beta_2$, respectively, are significant. For $\beta_2$, I would put the significance level at $\alpha/2=0.25$. That is because I am not testing $\beta_5$ before $\beta_2$ and we already dealt with $\beta_1$. That means at this point, I only considered $\beta_2$ and $\beta_4$. For $\beta_4$, I would put the significance level at $\alpha/2=0.25$. That is because I already dealt with $\beta_1$ and $\beta_2$ - that is now identical to the standard Holm-Bonferroni test. Lastly for $\beta_5$, the significance level is $\alpha/1=0.05$ as usual.

Thus in this example, I would accept T1 and T3 and reject T3 and T4 using the Holm-Bonferroni correction. Is my dealing with the interaction effects $\beta_4$ and $\beta_5$ correct?

NOTE 1: In my actual problem, I have two separate models $y=\beta_0 + \beta_1x_1 +\beta_3x_3+\beta_4x_1x_3 + \varepsilon $ and $y=\beta_0 + \beta_2x_2+\beta_3x_3+\beta_5x_2x_3 + \varepsilon $ fitted with two different datasets. But I don't think this makes any difference in this case.

NOTE 2: The question How to apply Bonferroni correction when including an interaction term? has some similarities to this one, but I am not sure whether the answers also apply to my question.

  • $\begingroup$ Were all these models built on the same data? $\endgroup$ – user2974951 Jan 21 '19 at 11:37
  • $\begingroup$ Model 1 and 3 use the same sample as do Model 2 and 4; i.e. there are two different datasets. $\endgroup$ – Tom Pape Jan 21 '19 at 11:52

So you have the following situation:

Test model 1 on data 1, if significant -> additionally test model 3 on data 1
Test model 2 on data 2, if significant -> additionally test model 4 on data 2

Because these two approaches use different data sets these are two different processes.
The answer will also depend on your objective / primary hypotheses.

If you decide to test both models (say model 1 and 3) regardless of results of model 1, then you would adjust all the p-values from both models together.
If, however, you decide to conduct additional tests based on previous results (if they are significant), this is generally what people call "exploratory analysis" and different rules apply here. You will find different answers here, but in general, you would not adjust the p-values from the second model, since these results would be only "exploratory" and not "confirmatory", but you would still adjust the p-values from the first model.

Also, I am not sure what you mean by $\alpha/2$, if you are using Bonferroni then you divide by $n$ the number of conducted tests not the number of models.

  • $\begingroup$ Thank you for your detailed answer. Yes, your understanding of the question is correct. $\endgroup$ – Tom Pape Jan 21 '19 at 15:09
  • $\begingroup$ Note that the number of models is the number of tests in my example (I will clarify this now in the question by calling it competing hypotheses). I am in the process of submitting a pre-registered trial with the above mentioned four hypotheses, what appears to me a confirmative design. In my project Model 1 tests for a cognitive bias and model 3 tests for a way to partially unbias. Clearly, without the bias one does not need to test whether the method for unbiasing works. Thus, would you now say that my reasoning makes sense or what would be the right adjusted significance levels? $\endgroup$ – Tom Pape Jan 21 '19 at 15:12
  • $\begingroup$ @TomPape I don't really follow, I don't know how you are testing for bias with a hypothesis test. Also I don't know what you mean by models "competing". $\endgroup$ – user2974951 Jan 22 '19 at 13:38
  • 1
    $\begingroup$ @TomPape I think you are overcomplicating things, it is not clear what you are trying to show with this analysis. Why not test both coefficients at the same time as opposed to doing sequential tests? If you are testing whether a coefficient is different or not from 0 you can do this all in one go. $\endgroup$ – user2974951 Jan 25 '19 at 13:53
  • 1
    $\begingroup$ @TomPape It is general advice to try and put all your hypotheses into one model, that is define them all beforehand, instead of doing a "if then" kind of nested procedure. The method that you proposed is sure to raise eyebrows and if you are trying to write this for a paper then any good statistical reviewer would question this choice. $\endgroup$ – user2974951 Jan 27 '19 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.