I have a dataset with four groups and 14 different dependent variables. Based on the answer given here, I think the best approach for my analytical question is to perform multiple one-way ANOVAs, followed up with Tukey's tests for the dependent variables that showed significant ANOVAs.

My question is, how should I perform my multiple hypothesis corrections? It seems clear that I should apply a multiple hypothesis correction to the ANOVAs before deciding to run the post-hoc tests. When I run my Tukey's tests should the number of hypotheses I'm correcting for include the ANOVAs that were run?

To lay out an example:

  1. I run my tests and find that 4 of my ANOVAs were significant after multiple hypothesis correction.
  2. I then run 4 Tukey tests and obtain 6 p-values from each (for each unique pairwise comparison between groups). I'm using TukeyHSD in R, so the p-values it reports have already been adjusted for the multiple comparisons within each DV.

I have two basic questions:

  • Is it appropriate to apply a Benjamini-Hochberg correction to the p-values reported by TukeyHSD? Or is there another correction that's objectively more appropriate for this situation?
  • Regardless of the correction I'm using, should the number of hypotheses tested only include the post-hoc tests (24 p-values) or should it include the ANOVAs as well (24 + 14 = 38 p-values)?

Thank you!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.