# Which family-wise correction is needed for one-sided t-tests in a randomized controlled trial with several independent measures across groups?

I conducted a randomized controlled experiment with groups $$A$$, $$B$$, $$C$$, and $$D$$, each containing around $$40$$ human participants who saw one of four variants of a website. Group $$A$$ saw a baseline, and the other groups saw that baseline with a single additional component: $$B$$ saw an extra slider to pick values, and $$C$$ and $$D$$ saw that slider with one of two additional components. The rationale of the experiment was to study how users' perceptions and behaviours were affected. Therefore, participants filled out $$4$$ Likert scales in a post-study questionnaire (measuring constructs such as trust), and I logged $$4$$ additional metrics (e.g., chosen slider values).

Now I want to test whether these $$8$$ measurements differ across groups. Based on theory and qualitative studies, my hypotheses were:

• For 4 measurements: $$\mu_A<\mu_B$$, $$\mu_B<\mu_C$$, and $$\mu_B<\mu_D$$;
• For 1 measurement: $$\mu_A=\mu_B$$, $$\mu_B<\mu_C$$, and $$\mu_B<\mu_D$$;
• For 1 measurement: $$\mu_A<\mu_B$$, $$\mu_B=\mu_C$$, and $$\mu_B=\mu_D$$;
• For 2 measurements: $$\mu_B<\mu_C$$, and $$\mu_B<\mu_D$$.

After checking assumptions, I applied one-sided $$t$$-tests and $$8$$ out of the $$19$$ tests were significant: some are of order $$p=0.01-0.03$$, others $$p<10^{-5}$$. Given these results and the domain-specific consensus that $$p<0.05$$ is significant, I have two questions:

1. Should I correct the $$p$$-values? I am aware of the debate about whether post-hoc correction methods such as Bonferroni are necessary in the first place (e.g., How many p-value observations do you think are required before doing FDR correction), but am unsure whether my experiment should be considered exploratory or confirmatory.
2. If necessary, how should I correct the $$p$$-values? It is unclear to me whether corrections are only required for multiple comparisons against the same baseline (e.g., $$\mu_B<\mu_C$$ and $$\mu_B<\mu_D$$). Furthermore, should I correct the $$p$$-values measurement-by-measurement or for all measurements at once?
• What do you mean they each have 40 samples? Do you mean each group has 40 participants? Commented Dec 12, 2023 at 2:07
• @ShawnHemelstrand Yes, I meant participants and have edited my question. Commented Dec 12, 2023 at 11:54
• Based on your hypotheses, you might have a look at both intersection -union tests and bioequivalence tests. Commented Dec 18, 2023 at 13:32
• Please explain what the measured variables are and what your groups represent. From your hypotheses it seems that the groups could be of ordinal nature? If this is an RCT, what is the treatment, how does it differ between groups?
– jkd
Commented Dec 31, 2023 at 11:08
• @jkd Thanks for following up. I have updated my question to explain the treatment and measurements better. Commented Jan 22 at 1:21