1
$\begingroup$

For my dataset, I have two groups.

For each group I would like to perform a one-sample t-test to compare the value against 0. I would then like to test a two-sample t-test to compare the mean of each group.

Should I correct for the multiple comparison for the one-sample tests only (i.e. alpha/2) or correct for all three multiple comparisons (alpha/3)?

My issue is that if one group is found to be significantly above 0 while the other isn't, then correcting the the third multiple comparison is likely to be too conservative.

And on a related note, I need to perform similar tests for another dataset, which was derived from an entirely different population. Should I now correct for all the comparisons that were performed on the first and second datasets?

$\endgroup$
1
$\begingroup$

Depends on how much correction you want to do and how conservative you want to be. Correcting for multiple testing across all the tests you've carried out is the most conservative option, but in lots of genomics experiments and papers, where we look at high dimensional data, for instance, it is a practise to correct for FDR within that experiment while not doing so for additional stats on low-dimensional data. Likewise if there are multiple high dimensional experiments one usually corrects for FDRs within the hypotheses tested on each dataset (for obvious reasons, high dimensional data can lead to declining significance globally very easily).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.