5
$\begingroup$

My experiment has a 2x2 design, full between-subjects, so that the resulting four groups are as follows: -Group A-1: Experimental treatment A. -Group A-2: Control for experimental treatment A. -Group B-1: Experimental treatment B. -Group B-2: Control for experimental treatment B.

I am expecting that treatment A works (when compared to their control) while treatment B doesn't. As the interaction is significant, I need to follow up with contrasts. In Jamovi and similar software, this is straightforward with Tukey's procedure.

However, note that not all the contrasts are useful here. In fact it makes no sense to compare the experimental treatment A with the control for the treatment B. Tukey just takes all possible combinations of the 4 means and correct accordingly. Here only two comparisons are of interest: Treatment A vs. Control A, and Treatment B vs. Control B. Tukey corrects for 6 comparisons even if most of them are useless.

I've been reading about other procedures to protect alpha from multiple comparisons. Bonferroni is the easiest to apply but overly conservative (right?). Dunnett allows to reduce the number of contrasts but assumes only one control.

If I am interested in only two contrasts (and the interaction that reveals that treatment A works better than treatment B), which correction procedure can I apply?

Improve this question
$\endgroup$
6
  • $\begingroup$ Have you considered using Holm's method? $\endgroup$
    – statmerkur
    Jun 15 at 10:37
  • $\begingroup$ I don't think that would work. Holm-Bonferroni is a sequential method and still assumes that I'm interested in all 6 contrasts, whereas only 2 of them make sense. $\endgroup$
    – Experimental Psychologist
    Jun 15 at 19:54
  • $\begingroup$ Why would the method assume that you're interested in all 6 contrasts? It controls the FWER strongly within a prespecified "family" of to be tested null hypotheses. $\endgroup$
    – statmerkur
    Jun 16 at 21:32
  • $\begingroup$ Oh, perhaps this is something I don't understand. But this B-H procedure (as all sequential methods) will start by picking up the two most different means, or the smaller p-value, then proceed to the next, etc. If I followed this procedure, what if the first three or four constast that I test are completely useless because they are comparing the two controls with each other, or one treatment with the control from the other condition... And this leaves me exactly where I was when I asked the question: Do I need to assume that all combinations of groups need to be compared, and correct for them? $\endgroup$
    – Experimental Psychologist
    Jun 17 at 8:47
  • $\begingroup$ If you only want to test two planned contrasts, it is sufficient to apply Holm's method to the two corresponding hypothesis tests. But you might want to do three tests if your second hypothesis is about equivalence (see my answer). $\endgroup$
    – statmerkur
    Jun 18 at 0:41

2 Answers 2

Reset to default
4
$\begingroup$

This is a good question.

In your situation, it's common and perfectly ok to just run two t tests for the contrasts of interest, and do the Boneferroni correction manually by multiplying the p values by 2. With only two contrasts, there will be very little difference between this and other, harder to justify corrections.

Improve this answer
$\endgroup$
3
$\begingroup$

In what follows I assume that you want to control the FWER (in the strong sense). In general, if you want to test a fixed number of arbitrary planned contrasts (in your case: treatment A vs. control A, treatment B vs. control B), the Holm method can be used to control the FWER strongly within the family of hypothesis tests for these contrasts. The Holm method is more powerful than the Bonferroni correction, which could also be applied in this setting.
Note that this is to be distinguished from the case where contrasts suggested by the data (e.g., the difference between the two group means that differ the most) are tested. Here Scheffé's method (among other, more powerful, methods tailored to specific types of comparisons) could be used. If the planned contrasts fully describe your hypotheses, there is also no need for an omnibus test.
Note also that your second hypothesis "treatment B doesn't work" suggests that you are looking for an equivalence test which, when based on two one-sided t-tests (TOST), would require rejecting two null hypotheses to support your (research) hypothesis.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.