I am performing multiple (around 40) multi-way ANOVAs on a dataset, one for each outcome variable. There are three factors involved: cohort, pre/post, and tissue type. For each outcome variable, I perform the following procedure:
- overall F-test for a model that includes all the factors
- multi-way ANOVA which gives me p-values for all main effects and all interaction effects
- for any effect that is significant, perform a post-hoc Tukey test to see which pairwise comparisons are significant
My question is the following: I realized just now that I really should adjust the overall F-tests for multiple comparisons using some kind of multiple testing correction, as I am doing ~40 ANOVAs. Do I need to adjust the p-values for the main effects/interaction effects as well? And do I need to adjust the p-values for the Tukey test again (even though it's already correcting for the multiple pairwise comparisons)?
Edited to add: there are multiple questions that appear to be a duplicate of this question (e.g. this or this) but none of them have answers. If there are duplicate questions that have answers, feel free to point me to them.