Partially orthogonal planned comparisons

Question

I am comparing 8 means and want to set up a planned comparisons, rather than having my Bonferroni adjustment become overly-conservative in a post-hoc. For my groups I need to make a total of 16 comparisons, and some of these comparisons are orthogonal to each other, while others are not. I am curious if there is a way to adjust my alpha without being penalized for all of my comparisons. Here's a summary of my comparisons:

Groups:     A    B    C    D    E    F    G    H
1:          1   -1    0    0    0    0    0    0
2:          0    0    1   -1    0    0    0    0
3:          0    0    0    0    1   -1    0    0 
4:          0    0    0    0    0    0    1   -1
5:          1    0   -1    0    0    0    0    0
6:          1    0    0    0   -1    0    0    0
7:          1    0    0    0    0    0   -1    0
8:          0    0    1    0   -1    0    0    0
9:          0    0    1    0    0    0   -1    0
10:         0    0    0    0    1    0   -1    0 
11:         0    1    0   -1    0    0    0    0
12:         0    1    0    0    0   -1    0    0
13:         0    1    0    0    0    0    0   -1
14:         0    0    0    1    0   -1    0    0
15:         0    0    0    1    0    0    0   -1
16:         0    0    0    0    0    1    0   -1

So, some comparisons (e.g. 1-4) are orthogonal to each other, while other comparisons are not. Each mean gets used for a total of 4 comparisons, so is it a stretch to assume an alpha of 0.05 needs to only be adjusted to 0.0125 to account for increased chance of type I error? Or is the rule more strict?

Answer 1

Unfortunately, I don't think it works that way. If you want to hold the familywise error rate to $.05$, then you need to adjust for more than $4$ comparisons. However, you can use a 'stepdown' (a.k.a., Holm's) procedure to gain some power back. Here's how it works:

1. Conduct an ANOVA over all $8$ groups. If this is significant, there is at least $1$ false null. Thus, you don't need to control for $16$ contrasts. The most extreme case is that $7$ of the groups have identical means, but $1$ group differs from the rest. Since each group is involved in $4$ contrasts, this implies that a significant overall ANOVA means there can be at most $12$ true nulls among your $16$ contrasts.
2. Run all of your contrasts and sort them into ascending order by $p$-value. You can begin assessing the first $5$ (lowest) $p$-values to see if they are lower than your adjusted $\alpha$ ($\alpha_{\rm adj}$, which is your nominal $\alpha/12$). You can call any of the $5$ most significant contrasts that have $p < \alpha_{\rm adj}$ significant.
3. If $5$ contrasts have been declared significant by this procedure, then there can be at most $11$ true nulls among the remaining set. Thus, your new $\alpha_{\rm adj}$ would be $\alpha/11$, which is a slightly easier threshold to cross, giving you slightly more power. It the sixth lowest $p$-value is less than this threshold, you call it significant, and conclude that there can be at most $10$ true nulls among your set of remaining contrasts.
4. So your next lowest $p$-value is compared to $\alpha_{\rm adj} = \alpha/10$. Etc.

At any point, if the next lowest $p$-value is $> \alpha_{\rm adj}$, you stop.

This is true even if you need to stop within the first $4$ contrasts. In which case logically, either the ANOVA was a type I error, or there are some type II errors amongst your non-significant contrasts. Nonetheless, the status of the remaining contrasts is ambiguous. Remember that the fact a test is non-significant does not mean there is no difference, only that you cannot tell with the desired level of confidence. (For more on this, it may help to read my answer here: Why do statisticians say a non-significant result means "you cannot reject the null" as opposed to accepting the null hypothesis?)