I conducted a randomized controlled experiment with groups $A$, $B$, $C$, and $D$, each containing around $40$ human participants who saw one of four variants of a website. Group $A$ saw a baseline, and the other groups saw that baseline with a single additional component: $B$ saw an extra slider to pick values, and $C$ and $D$ saw that slider with one of two additional components. The rationale of the experiment was to study how users' perceptions and behaviours were affected. Therefore, participants filled out $4$ Likert scales in a post-study questionnaire (measuring constructs such as trust), and I logged $4$ additional metrics (e.g., chosen slider values).

Now I want to test whether these $8$ measurements differ across groups. Based on theory and qualitative studies, my hypotheses were:

  • For 4 measurements: $\mu_A<\mu_B$, $\mu_B<\mu_C$, and $\mu_B<\mu_D$;
  • For 1 measurement: $\mu_A=\mu_B$, $\mu_B<\mu_C$, and $\mu_B<\mu_D$;
  • For 1 measurement: $\mu_A<\mu_B$, $\mu_B=\mu_C$, and $\mu_B=\mu_D$;
  • For 2 measurements: $\mu_B<\mu_C$, and $\mu_B<\mu_D$.

After checking assumptions, I applied one-sided $t$-tests and $8$ out of the $19$ tests were significant: some are of order $p=0.01-0.03$, others $p<10^{-5}$. Given these results and the domain-specific consensus that $p<0.05$ is significant, I have two questions:

  1. Should I correct the $p$-values? I am aware of the debate about whether post-hoc correction methods such as Bonferroni are necessary in the first place (e.g., How many p-value observations do you think are required before doing FDR correction), but am unsure whether my experiment should be considered exploratory or confirmatory.
  2. If necessary, how should I correct the $p$-values? It is unclear to me whether corrections are only required for multiple comparisons against the same baseline (e.g., $\mu_B<\mu_C$ and $\mu_B<\mu_D$). Furthermore, should I correct the $p$-values measurement-by-measurement or for all measurements at once?
  • $\begingroup$ What do you mean they each have 40 samples? Do you mean each group has 40 participants? $\endgroup$ Commented Dec 12, 2023 at 2:07
  • $\begingroup$ @ShawnHemelstrand Yes, I meant participants and have edited my question. $\endgroup$
    – Jeroen
    Commented Dec 12, 2023 at 11:54
  • $\begingroup$ Based on your hypotheses, you might have a look at both intersection -union tests and bioequivalence tests. $\endgroup$ Commented Dec 18, 2023 at 13:32
  • $\begingroup$ Please explain what the measured variables are and what your groups represent. From your hypotheses it seems that the groups could be of ordinal nature? If this is an RCT, what is the treatment, how does it differ between groups? $\endgroup$
    – jkd
    Commented Dec 31, 2023 at 11:08
  • $\begingroup$ @jkd Thanks for following up. I have updated my question to explain the treatment and measurements better. $\endgroup$
    – Jeroen
    Commented Jan 22 at 1:21

1 Answer 1


You do not supply any context for this study and so it is not possible to even say if multiple comparisons 'corrections' are relevant to your inferential objectives.

Questions about multiple comparisons 'corrections' appear on this site all the time and so there are many answers that are likely to help you already in place. I, of course, prefer my own answers so I suggest that you have a look at these: How many p-value observations do you think are required before doing FDR correction

p value correction in multiple outcomes study

  • $\begingroup$ Thanks for your answer, Michael. What kind of extra context would be necessary? $\endgroup$
    – Jeroen
    Commented Dec 12, 2023 at 11:56
  • 1
    $\begingroup$ Context that you will be able to glean from the answers that I've linked. $\endgroup$ Commented Dec 12, 2023 at 19:43
  • $\begingroup$ Sorry for the delay and thanks for following up @Michael Lew. I have updated my question and hope it now provides enough details. I read your previous answers and your chapter "A Reckless Guide to P-values". $\endgroup$
    – Jeroen
    Commented Jan 22 at 1:19
  • 1
    $\begingroup$ It sounds like a preliminary, hypothesis generating study, so do not do any 'correction' for multiplicity. Instead, discuss the results and say which of the supported hypotheses are most worthy of follow-up experimentation. $\endgroup$ Commented Jan 22 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.