Bonferroni and Greenhouse-Geisser corrections with repeated-measures ANOVA

Question

I guess my question comes in two parts:

1. Following a 1x7 repeated-measures ANOVA, I carried out 3 pre-planned tests. Two were t-tests between 2 of the conditions. The third is a 1x6 ANOVA (to test significance between conditions 2-7). My question is:
   Do I have to apply the Bonferroni correction to all 3 tests, or just the 2 tests? I.e. should the significance level be 0.05/3 = 0.017 for all 3 tests?

2. As above, I carried out a 1x7 repeated-measures ANOVA, and 3 pre-planned tests (2 t-tests and a 1x6 ANOVA). In the original 1x7, sphericity was assumed, so I report the 'sphericity-assumed' statistic, and it is non-significant. Next, despite this non-significance, I decided I should do the pre-planned tests for the sake of completeness. When I do the 1x6 repeated measures ANOVA, sphericity is NOT assumed, so I use the Greenhouse-Geisser values, and it comes up significant (if I were to use the shericity assumed values, it would be non-significant). My problem here is that the ANOVA using just 6 of the 1x7 conditions is coming up significant when the original 1x7 was non-significant. Logically, this doesn't seem to make sense to me.
   Is there some 'rule of thumb' whereby the use of Greenhouse-Geisser values needs to be consistent?

I hope this makes sense. I have trawled the internet. I have looked at more stats books than you can shake a stick at. But nothing comes close to discussing this sort of situation.

Answer 1

Regarding Bonferroni (and multiple comparisons issues in general) Jacob Cohen, in his book on regression, said "this is a subject on which reasonable people can differ". There are arguments for not doing such corrections at all (see, e.g., this piece by Andrew Gelman). I find such arguments persuasive.

If you reduce chance of type 1 error then (other things being equal) you increase chance of type 2 error (that is, you reduce power). In many fields, the "given" values are .05 for type 1 and .8 for power (.2 for type 2). But these may or may not be sensible. Type 2 error may be much worse than type 1 error.

In my view, the important thing is not statistical significance, and certainly not whether it is below .05, but effect size and precision of effect size.

If you correct for multiple comparisons, it is really a question of what "multiple" is. Paraphrasing Cohen, should it be for one experiment? One article? One set of results? Results about one topic? All results you ever do? Or what?

From a purely practical point of view, you may be required to do some particular thing to satisfy journal editors or other supervisors.

Answer 2

A couple of points - too long for a comment:

1. If you are protecting your pre-planned tests with Bonferroni correction, than there is no need to run the original ANOVA. The "double protection" only looses power.
2. Most of the standard "post-hoc" tests for ANOVA do not need the protection of the original F-test. Essentially the only approach that needs it is when the post-hoc tests are not adjusted at all. This is usually not recommended, because this approach only provides weak control of the familywise type I error rate (i.e. the error rate of the pairwise comparisons is controlled only if all the means are equal)
3. You are doing strange things with the sphericity assumption. Approximately, it means that any pairwise differences have the same variance. If you are willing to assume that the entire design satisfies it, then any subset should as well. So if you are using GG correction for the subset, you should be using it for the entire set (the reverse does not have to be true). If the GG correction changes the results substantially (not just moving the p-value from 0.048 to 0.052), then the sphericity is probably not satisfied.
4. You keep confusing "not significant" with "there is no difference". While this is tempting to do, this thinking leads to all sort of apparent paradoxes.