2
$\begingroup$

I am performing multiple (around 40) multi-way ANOVAs on a dataset, one for each outcome variable. There are three factors involved: cohort, pre/post, and tissue type. For each outcome variable, I perform the following procedure:

  1. overall F-test for a model that includes all the factors
  2. multi-way ANOVA which gives me p-values for all main effects and all interaction effects
  3. for any effect that is significant, perform a post-hoc Tukey test to see which pairwise comparisons are significant

My question is the following: I realized just now that I really should adjust the overall F-tests for multiple comparisons using some kind of multiple testing correction, as I am doing ~40 ANOVAs. Do I need to adjust the p-values for the main effects/interaction effects as well? And do I need to adjust the p-values for the Tukey test again (even though it's already correcting for the multiple pairwise comparisons)?

Edited to add: there are multiple questions that appear to be a duplicate of this question (e.g. this or this) but none of them have answers. If there are duplicate questions that have answers, feel free to point me to them.

Improve this question
$\endgroup$
4
  • $\begingroup$ So in each ANOVA you use a DV, which is always different, and all the IV's of the data set? $\endgroup$
    – user2974951
    Oct 26 '18 at 5:09
  • $\begingroup$ Yes, that’s exactly it. $\endgroup$
    – abr
    Oct 28 '18 at 4:19
  • 1
    $\begingroup$ 40 ANOVA is extreme... ideally you would want to adjust everything. Although it is worth noting that the first 2 points are based on the F-test results and the third point is based on the t-test results, so different tests. Personally I would separately adjust all the p-values from the the first 2 points from the p-values from the last point. Note: some Tukey post-hoc tests automatically perform p-value adjustment, so I would calculate the un-adjusted p-values, and then correct them all together (all the tukey results). $\endgroup$
    – user2974951
    Oct 28 '18 at 14:13
  • $\begingroup$ @user2974951 If you make your response an answer then I can accept it as it has been helpful. A follow-up question, though - when you adjust the p-values for main effect/interaction effect would you adjust them all at the same time (so ~6 p-values per ANOVA * 40 ANOVAs would become 240 p-values to adjust simultaneously) or adjust the p-value for each effect separately (i.e. adjust 40 p-values six times). $\endgroup$
    – abr
    Oct 30 '18 at 16:32
1
$\begingroup$

Although many similar questions have been asked on this site I have not been able to find a good or conclusive answer on the matter. Answers vary from correcting everything to correcting only within the levels of testing (so in your ex. correcting all the p-values that are in the same point). Also general advice is to consider reducing the number of tests or considering a different approach. Maybe you have some domain knowledge which allows you to drop some tests because these are probably unnecessary. Performing multiple testing adjustment for 40 ANOVA's is going to significantly affect your p-values, this should be done only if you have very high power.

Nevertheless, if I had to give an answer on the matter (my statistical opinion), I would say to correct all the p-values from the first two points together, so 40*6 values, that is 40 ANOVA's with 6 terms each. My reasoning: these values all come from the same underlying test, the F-test. In the case of the overall ANOVA test, this is an F-test of the whole model variance compared to the null model variance. In the case of the model term test, this is an F-test for the ANOVA model variance with this term added compared to the ANOVA model variance without this term. So basically these are equal tests, done for different purposes.

The Tukey post-hoc test I would adjust them separately, all of them together, since these results come from a different test, the t-test. Note: some Tukey post-hoc tests automatically perform p-value adjustment, so I would calculate the un-adjusted p-values, and then correct them all together (in the Tukey post-hoc manner).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.