Following one-way ANOVA, there are many possible follow-up multiple comparison tests. Holm's test (or better, the Holm-Sidak) test has lots of power, but because it works in a stepwise manner, it cannot compute confidence intervals. Its advantage over the tests than can compute confidence intervals (Tukey, Dunnett) is that is has more power. But is it fair to say that the Holm method always has more power than the methods of Tukey and Dunnet? Or does it depend...?
closed as not a real question by csgillespie, Peter Smit, mbq♦, Srikant Vadali Aug 5 '10 at 13:06
It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.
I usually like to run simulations to answer questions like this, but without confirmed details of the algorithm the question asker wants evaluated and no obvious implementation of the Holm-Sidak procedure available in R, that is not possible. For my answer I eyeballed the code provided here. Assuming that is the right procedure, and assuming the null hypothesis is that all group means are equal:
Feel free to correct me. I usually like to run simulations on such things. But, without the ability to readily do that, I can't check my answer. So, I might be entirely wrong here. My answer is that usually the Holm-Sidak will demonstrate greater power, but that the answer in a strict sense is "it depends". Both methods use the pooled error term and assume homogeneity of variance, so there is no difference in the procedures there. However, since Holm-Sidak adjusts in a stepwise manner early comparisons are more likely to pass a threshold of significance than later comparisons, especially since the freedom to assess later comparisons is dependent on the outcome of previous comparisons. Thus, it seems likely that in situations where the differences in means between groups to be compared is roughly equal (and meets the Tukey HSD threshold for significance) and the number of groups is sufficiently large (purposely vague without a simulation), that Holm-Sidak will fail to reach significance for the later comparisons. Thus, in these situations Tukey's HSD will have more power than Holm-Sidak.