Adjusting p-values for multiple ANOVAs I am performing multiple (around 40) multi-way ANOVAs on a dataset, one for each outcome variable.  There are three factors involved: cohort, pre/post, and tissue type.  For each outcome variable, I perform the following procedure:


*

*overall F-test for a model that includes all the factors

*multi-way ANOVA which gives me p-values for all main effects and all interaction effects

*for any effect that is significant, perform a post-hoc Tukey test to see which pairwise comparisons are significant


My question is the following: I realized just now that I really should adjust the overall F-tests for multiple comparisons using some kind of multiple testing correction, as I am doing ~40 ANOVAs.  Do I need to adjust the p-values for the main effects/interaction effects as well? And do I need to adjust the p-values for the Tukey test again (even though it's already correcting for the multiple pairwise comparisons)?
Edited to add: there are multiple questions that appear to be a duplicate of this question (e.g. this or this) but none of them have answers.  If there are duplicate questions that have answers, feel free to point me to them.
 A: Although many similar questions have been asked on this site I have not been able to find a good or conclusive answer on the matter. Answers vary from correcting everything to correcting only within the levels of testing (so in your ex. correcting all the p-values that are in the same point). Also general advice is to consider reducing the number of tests or considering a different approach. Maybe you have some domain knowledge which allows you to drop some tests because these are probably unnecessary. Performing multiple testing adjustment for 40 ANOVA's is going to significantly affect your p-values, this should be done only if you have very high power.
Nevertheless, if I had to give an answer on the matter (my statistical opinion), I would say to correct all the p-values from the first two points together, so 40*6 values, that is 40 ANOVA's with 6 terms each. My reasoning: these values all come from the same underlying test, the F-test. In the case of the overall ANOVA test, this is an F-test of the whole model variance compared to the null model variance. In the case of the model term test, this is an F-test for the ANOVA model variance with this term added compared to the ANOVA model variance without this term. So basically these are equal tests, done for different purposes.
The Tukey post-hoc test I would adjust them separately, all of them together, since these results come from a different test, the t-test. Note: some Tukey post-hoc tests automatically perform p-value adjustment, so I would calculate the un-adjusted p-values, and then correct them all together (in the Tukey post-hoc manner).
