I guess my question comes in two parts:

  1. Following a 1x7 repeated-measures ANOVA, I carried out 3 pre-planned tests. Two were t-tests between 2 of the conditions. The third is a 1x6 ANOVA (to test significance between conditions 2-7). My question is:
    Do I have to apply the Bonferroni correction to all 3 tests, or just the 2 tests? I.e. should the significance level be 0.05/3 = 0.017 for all 3 tests?

  2. As above, I carried out a 1x7 repeated-measures ANOVA, and 3 pre-planned tests (2 t-tests and a 1x6 ANOVA). In the original 1x7, sphericity was assumed, so I report the 'sphericity-assumed' statistic, and it is non-significant. Next, despite this non-significance, I decided I should do the pre-planned tests for the sake of completeness. When I do the 1x6 repeated measures ANOVA, sphericity is NOT assumed, so I use the Greenhouse-Geisser values, and it comes up significant (if I were to use the shericity assumed values, it would be non-significant). My problem here is that the ANOVA using just 6 of the 1x7 conditions is coming up significant when the original 1x7 was non-significant. Logically, this doesn't seem to make sense to me.
    Is there some 'rule of thumb' whereby the use of Greenhouse-Geisser values needs to be consistent?

I hope this makes sense. I have trawled the internet. I have looked at more stats books than you can shake a stick at. But nothing comes close to discussing this sort of situation.

  • $\begingroup$ Re: your first question, multiple comparisons is a frequently discussed topic on this site, so I'd suggest do a search of the old threads. I can't comment directly on your second question (re: sphericity), but it certainly is possible, in ANOVA, to reject the null in the first case, but not reject the null when you've added a new level - particularly if that new level has a mean equal to, say, the mean of the other 6 levels. $\endgroup$ – Macro Apr 26 '12 at 12:12
  • $\begingroup$ Re: Marco... I see your point about adding a new level bringing significance to non-significance, but in my data, removing a level turned non-significance to significance... $\endgroup$ – Eric Apr 26 '12 at 12:53
  • $\begingroup$ Maybe the level that was removed had a mean very similar to the others so, by removing it, there was more heterogeneity in the means? $\endgroup$ – Macro Apr 26 '12 at 12:55

Regarding Bonferroni (and multiple comparisons issues in general) Jacob Cohen, in his book on regression, said "this is a subject on which reasonable people can differ". There are arguments for not doing such corrections at all (see, e.g., this piece by Andrew Gelman). I find such arguments persuasive.

If you reduce chance of type 1 error then (other things being equal) you increase chance of type 2 error (that is, you reduce power). In many fields, the "given" values are .05 for type 1 and .8 for power (.2 for type 2). But these may or may not be sensible. Type 2 error may be much worse than type 1 error.

In my view, the important thing is not statistical significance, and certainly not whether it is below .05, but effect size and precision of effect size.

If you correct for multiple comparisons, it is really a question of what "multiple" is. Paraphrasing Cohen, should it be for one experiment? One article? One set of results? Results about one topic? All results you ever do? Or what?

From a purely practical point of view, you may be required to do some particular thing to satisfy journal editors or other supervisors.


A couple of points - too long for a comment:

  1. If you are protecting your pre-planned tests with Bonferroni correction, than there is no need to run the original ANOVA. The "double protection" only looses power.
  2. Most of the standard "post-hoc" tests for ANOVA do not need the protection of the original F-test. Essentially the only approach that needs it is when the post-hoc tests are not adjusted at all. This is usually not recommended, because this approach only provides weak control of the familywise type I error rate (i.e. the error rate of the pairwise comparisons is controlled only if all the means are equal)
  3. You are doing strange things with the sphericity assumption. Approximately, it means that any pairwise differences have the same variance. If you are willing to assume that the entire design satisfies it, then any subset should as well. So if you are using GG correction for the subset, you should be using it for the entire set (the reverse does not have to be true). If the GG correction changes the results substantially (not just moving the p-value from 0.048 to 0.052), then the sphericity is probably not satisfied.
  4. You keep confusing "not significant" with "there is no difference". While this is tempting to do, this thinking leads to all sort of apparent paradoxes.
  • $\begingroup$ Thank for your comments. With regard to point 3 - the reason I did not assume sphericity for the entire design is because Mauchly's for the 1x7 is less than .05. It is only removing one condition which then creates a subset of data where sphericity becomes assumed (Mauchly's for the 1x6 is >.05), and the p value is .039. If G-G was used, it would be .076. Would you advise to just report the 1x7 as non-significant, sphericity not assumed, and the 1x6 significant, sphericity assumed? And then just try and make sense of why this happening with regards to to the experiment? $\endgroup$ – Eric Apr 27 '12 at 18:06
  • $\begingroup$ I don't know enough about the power of Mauchly's test to tell whether it is good enough at picking up deviations that affect the validity of the p-value. The difference between the corrected and uncorrected p-values are tiny, and we are only talking about it because the magic threshold got crossed. I would stick with the same assumptions for the subset as for the whole set. $\endgroup$ – Aniko Apr 27 '12 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.