Normalization in pairwise hypothesis testing I'm reviewing a conference paper (not a stats conference). The authors have done some funny things with their methodology. The experiment setup has 4 treatments, each applied to the same 30 subjects. This is repeated in 10 different circumstances.
ANOVA is used to determine whether the choice of treatment has any effect, separately for each of the 10 different circumstances. They find that only one circumstance had a significant difference, at p=0.03.
I have two questions regarding their methodology:


*

*It seems to me that the circumstances should be used as a separate factor in ANOVA, rather than doing 10 separate ANOVAS, and that if one did the test 10 times, one should multiply the p-values by 10 (i.e. a Bonferroni correction). Does this make sense?

*In spite of their earlier results, the authors then did pairwise t-tests between all the different treatment groups in all 10 circumstances. They report a statistically significant difference in means between many of the groups in circumstances which ANOVA suggested had no differences. I suspect this means they are not doing a Bonferroni correction for the p-values in their pairwise tests. Is this correct? Is there another way that ANOVA could suggest no significance, but a a significant difference could exist between the groups?
 A: *

*Yes, they should do multiple testing correction for trying to show the differences in 10 different conditions. This is assuming that they tried 10 conditions, with the starting hypothesis that some of them may show a difference. If the starting (a priori) hypothesis of their experiments was that this one particular condition will show a difference, and the rest won't, then they don't need to do multiple testing corrections.

*Yes, multiple pairwise t-tests between different treatments are too liberal when the correct test is ANOVA. For 4 treatments, the Bonferroni correction would be dividing by $4*3/2=6$.

A: Whether you should combine 10 different ANOVA's into a single ANOVA depends on what the scientific question is.  Remember there are assumptions that go into ANOVA models, one is equal variances, if the 10 different conditions lead to widely different variances then combining them into a single analysis may violate the assumptions.  On the other hand if all the conditions are similar and hypotheses about how they relate to each other are of interest then combining them would make the most sense.
If the 10 conditions are different enough and each is interesting in its own right then the seperate analyses are probably meaningful without correction, after all no statistician estimates the total number of tests they will do throughout their life and multiplies all p-values by that number.  But if the researchers will declare success if any of the individual Anovas is significant then a correction is appropriate.
For number 2, doing all pairwise tests after a non-significant ANOVA is pretty much always data dredging, or torturing the data until it confesses (and the fact that they did that suggests that they may have not thought through the 1st part well enough either).
