In the book Multiple Comparisons Using R Bretz et al write that their R package multcomp improves power above other multiple comparison procedures by "accounting for the structural correlations between the test statistics" (pg 41). The default option for this multiple comparison procedure implements this and is called "single-step."
My questions are
1)What do they mean by "structural correlations" (their explanation eludes me)
2)Is their a standard way to refer to or cite this approach. "Single-step" is what it is called in the call to their R function, but this is a general description of multiple comparisons.
In my field people are mostly familiar with the usual Bonferroni corrections and I'd like to avoid the critical eye of reviewers and also point readers toward work that can help them understand what is going on.
2 Answers
The Bonferroni procedure works for the worst case dependency between the involved single test statistics which is somehow negatively correlated. In contrast, the Sidák procedure works for independency between them. Clearly: You don't need to adjust if the multiple tests are copies from each other. Conversely, you have to adjust more, if one test tends to decide the opposite of the other one in order to meet the FWER.
In the special case of linear test statistics in normally distributed random variables (t-test), the dependency can pretty well be decribed by the correlation of the effect estimators. E.g. if you test simultaneously $ \mu_1 - \mu_2$ and $\mu_1 - \mu_3$, your test statistics will base on the respective means: $\bar{x}_1 - \bar{x}_2$ and $\bar{x}_1 - \bar{x}_3$. Both differences are correlated due to the structure of the hypotheses. Their covariance is the variance of $ \bar{x}_1$, and so the test statistics will likewise be (asyptotically, i.e. up to studentization) correlated. That's why Sidák cannot be used. But if you incorporate the dependency as Bretz et al. suggest, you will meet your FWER(strong) less conservative than if you would use Bonferroni.
Single step means that the local alpha will be calculated for all tests instateously. You can use these local alphas for simultaneous confidence intervals that convey the same decision as the tests would do. Stepwise (like Holm) means, that the local alphas depend on preceding test decisions. This means that the simultaneous confidence intervals would alsodepend on single test decisions! But confidence intervals are supposed to give information about their respective parameters, not about So if you have to supply confidence intervals, stepwise procedures are a bad choice.
-
$\begingroup$ 2 things: First, high positive correlation between tests doesn't usually mean the tests are exact "copies" of each other that don't require adjustment for multiple testing--in fact, there would be no point in doing multiple identical tests in the first place. Second, the Sidák procedure doesn't require independence--it only requires that there not be negative dependence. $\endgroup$ Commented May 12, 2017 at 8:58
This chapter is about the cases when some additional assumptions about the distribution of data are satisfied. For example, in linear regression setting due to the normality of errors the usial t-statistics for testing the significance of individual predictors not only have the univariate t-distribution each, but are jointly distributed according to the multivariate t (so their correlation could be explicitly estimated). I guess you could refer to this approach as a "parametric multiple testing procedure for linear regression" (or ANOVA, or whatever you are using). I don't think it has a more formal name.
