0
$\begingroup$

I have done a two-way repeated-measures ANOVA with factor A having 3 levels and factor B having 5 levels. I am doing post hoc comparisons between all 3 levels of factor A within each of the 5 levels of factor B, resulting in 15 comparisons. I am not explicitly doing any post hoc comparisons between the levels of factor B.

Assuming I use Bonferroni to correct for alpha inflation (disregarding whether or not it is the best correction method) would I need to correct by the number of tests I am actually doing (15), or the number of all possible comparisons, i.e. 45, including the 30 comparisons between the levels of factor B within each level of factor A? I have seen different opinions on which of these comparisons should be regarded as a family.

Thanks in advance!

$\endgroup$
0
$\begingroup$

no, you only correct for the ones you actually test. Basically, the correction is only to punish "hypothesis fishing". If you would correct for all possible ones, data sets with many variables or groups would always deal with underpowerdness, because their alpha would be lower just by having a large range of possible comparisons.

$\endgroup$
  • $\begingroup$ Thanks for your response! That I understand. However, as I understand it, post hoc tests are usually corrected exactly because they are somewhat a fishing expedition, as they are not hypothesis-guided, like planned comparisons. So there's, by definition, no specific reason to carry out only a subset of all possible comparisons, so that we should act as if we would do all of them (even if we don't di them explicitly). Where do I err? $\endgroup$ – JDsallin Dec 5 '18 at 16:04

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.