I conducted a randomized controlled experiment with groups $A$, $B$, $C$, and $D$, each containing around $40$ human participants who saw one of four variants of a website. Group $A$ saw a baseline, and the other groups saw that baseline with a single additional component: $B$ saw an extra slider to pick values, and $C$ and $D$ saw that slider with one of two additional components. The rationale of the experiment was to study how users' perceptions and behaviours were affected. Therefore, participants filled out $4$ Likert scales in a post-study questionnaire (measuring constructs such as trust), and I logged $4$ additional metrics (e.g., chosen slider values).
Now I want to test whether these $8$ measurements differ across groups. Based on theory and qualitative studies, my hypotheses were:
- For 4 measurements: $\mu_A<\mu_B$, $\mu_B<\mu_C$, and $\mu_B<\mu_D$;
- For 1 measurement: $\mu_A=\mu_B$, $\mu_B<\mu_C$, and $\mu_B<\mu_D$;
- For 1 measurement: $\mu_A<\mu_B$, $\mu_B=\mu_C$, and $\mu_B=\mu_D$;
- For 2 measurements: $\mu_B<\mu_C$, and $\mu_B<\mu_D$.
After checking assumptions, I applied one-sided $t$-tests and $8$ out of the $19$ tests were significant: some are of order $p=0.01-0.03$, others $p<10^{-5}$. Given these results and the domain-specific consensus that $p<0.05$ is significant, I have two questions:
- Should I correct the $p$-values? I am aware of the debate about whether post-hoc correction methods such as Bonferroni are necessary in the first place (e.g., How many p-value observations do you think are required before doing FDR correction), but am unsure whether my experiment should be considered exploratory or confirmatory.
- If necessary, how should I correct the $p$-values? It is unclear to me whether corrections are only required for multiple comparisons against the same baseline (e.g., $\mu_B<\mu_C$ and $\mu_B<\mu_D$). Furthermore, should I correct the $p$-values measurement-by-measurement or for all measurements at once?
