Post hoc contrasts when only certain contrasts make sense My experiment has a 2x2 design, full between-subjects, so that the resulting four groups are as follows:
-Group A-1: Experimental treatment A.
-Group A-2: Control for experimental treatment A.
-Group B-1: Experimental treatment B.
-Group B-2: Control for experimental treatment B.
I am expecting that treatment A works (when compared to their control) while treatment B doesn't.
As the interaction is significant, I need to follow up with contrasts. In Jamovi and similar software, this is straightforward with Tukey's procedure.
However, note that not all the contrasts are useful here. In fact it makes no sense to compare the experimental treatment A with the control for the treatment B. Tukey just takes all possible combinations of the 4 means and correct accordingly.
Here only two comparisons are of interest: Treatment A vs. Control A, and Treatment B vs. Control B. Tukey corrects for 6 comparisons even if most of them are useless.
I've been reading about other procedures to protect alpha from multiple comparisons. Bonferroni is the easiest to apply but overly conservative (right?). Dunnett allows to reduce the number of contrasts but assumes only one control.
If I am interested in only two contrasts (and the interaction that reveals that treatment A works better than treatment B), which correction procedure can I apply?
 A: In what follows I assume that you want to control the FWER (in the strong sense). In general, if you want to test a fixed number of arbitrary planned contrasts (in your case: treatment A vs. control A, treatment B vs. control B), the Holm method can be used to control the FWER strongly within the family of hypothesis tests for these contrasts. The Holm method is more powerful than the Bonferroni correction, which could also be applied in this setting.
Note that this is to be distinguished from the case where contrasts suggested by the data (e.g., the difference between the two group means that differ the most) are tested. Here Scheffé's method (among other, more powerful, methods tailored to specific types of comparisons) could be used.
If the planned contrasts fully describe your hypotheses, there is also no need for an omnibus test.
Note also that your second hypothesis "treatment B doesn't work" suggests that you are looking for an equivalence test which, when based on two one-sided t-tests (TOST), would require rejecting two null hypotheses to support your (research) hypothesis.
A: This is a good question.
In your situation, it's common and perfectly ok to just run two t tests for the contrasts of interest, and do the Boneferroni correction manually by multiplying the p values by 2. With only two contrasts, there will be very little difference between this and other, harder to justify corrections.
