2
$\begingroup$

I have multiple sets of comparisons I am running together. Normally I would do an FDR correction on the p-values to control the family-wise error rate. However, the p-values from a Tukey HSD are already adjusted for multiple comparisons (within a single test).

Does this pose problems for adjusting the p-values again?

As an example, let's say I am testing the difference among 3 treatments, first in group A (one test), and separately in group B (the second test). This gives me a total of 6 pairwise comparisons, 3 in each group. Normally I would adjust the resulting 6 p-values. But with Tukey HSD, the p-values have already been adjusted within each group (accounting for 3 comparisons), so to correct all 6 p-values as if they have not already been adjusted seems not quite right.

*EDIT: I neglected to say, this is after an omnibus ANOVA as well as simple effects test (because of significant interactions), both with corrected p-values. Having partitioned within levels of one factor, I proceed to Tukey's to examine combinations of the remaining factor.

$\endgroup$
1
  • $\begingroup$ If the interaction is significant, you can apply Tukey's HSD on the interaction term as well, right? $\endgroup$
    – chl
    Dec 8, 2020 at 19:19

2 Answers 2

1
$\begingroup$

Things I would suggest:

  1. Before running any a posteriori test such as Tukey's HSD, compute a p-value using an omnibus test, which will tell you if there are significant differences among any of the pairwise comparisons.
  2. IF the omnibus test indicates evidence that significant differences exist (e.g., p < 0.05), ensure that your data meet a criterion for homogeneity of variance (e.g., Levene's test).
  3. If the data meet your standard for homogeneity (e.g., p > 0.05), then use Tukey's HSD test to determine which parwise comparisons are specific and report an unadjusted p-value.

With this said, I think it is far more reasonable (and maybe preferable considering the general opinion of p-values, but if your boss/advisor/colleague/etc. wants p-values you may have to simply give them to him/her) is simply to report the omnibus p-value along with the group means and standard deviations. Any reader capable of interpreting the individual means should be able to tell which groups differ by a margin of any practical significance.

$\endgroup$
1
$\begingroup$

Always consider the nature of your scientific inferences and the role of the statistical inferences in those before adjusting or "correcting" any p-value for multiple testing. No advice on dealing with multiple comparisons is useful if it is not based on an understanding of what the experiments (or observations) are and what inferences might be made and for what purpose those inferences are intended.

In many circumstances there is no need to perform any formal adjustment for multiple tests, and even when such a procedure is desirable it is important to distinguish between the evidential meaning of the p-values and the error-rate properties of the statistical method.

See the multiple comparisons section of this open access chapter https://link.springer.com/chapter/10.1007/164_2019_286

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.