# Post hoc contrasts when only certain contrasts make sense

My experiment has a 2x2 design, full between-subjects, so that the resulting four groups are as follows: -Group A-1: Experimental treatment A. -Group A-2: Control for experimental treatment A. -Group B-1: Experimental treatment B. -Group B-2: Control for experimental treatment B.

I am expecting that treatment A works (when compared to their control) while treatment B doesn't. As the interaction is significant, I need to follow up with contrasts. In Jamovi and similar software, this is straightforward with Tukey's procedure.

However, note that not all the contrasts are useful here. In fact it makes no sense to compare the experimental treatment A with the control for the treatment B. Tukey just takes all possible combinations of the 4 means and correct accordingly. Here only two comparisons are of interest: Treatment A vs. Control A, and Treatment B vs. Control B. Tukey corrects for 6 comparisons even if most of them are useless.

I've been reading about other procedures to protect alpha from multiple comparisons. Bonferroni is the easiest to apply but overly conservative (right?). Dunnett allows to reduce the number of contrasts but assumes only one control.

If I am interested in only two contrasts (and the interaction that reveals that treatment A works better than treatment B), which correction procedure can I apply?

• Have you considered using Holm's method? Jun 15 at 10:37
• I don't think that would work. Holm-Bonferroni is a sequential method and still assumes that I'm interested in all 6 contrasts, whereas only 2 of them make sense. Jun 15 at 19:54
• Why would the method assume that you're interested in all 6 contrasts? It controls the FWER strongly within a prespecified "family" of to be tested null hypotheses. Jun 16 at 21:32
• Oh, perhaps this is something I don't understand. But this B-H procedure (as all sequential methods) will start by picking up the two most different means, or the smaller p-value, then proceed to the next, etc. If I followed this procedure, what if the first three or four constast that I test are completely useless because they are comparing the two controls with each other, or one treatment with the control from the other condition... And this leaves me exactly where I was when I asked the question: Do I need to assume that all combinations of groups need to be compared, and correct for them? Jun 17 at 8:47
• If you only want to test two planned contrasts, it is sufficient to apply Holm's method to the two corresponding hypothesis tests. But you might want to do three tests if your second hypothesis is about equivalence (see my answer). Jun 18 at 0:41