I have a dataset with one factor consisting of four levels. I've run a one-way repeated measures ANOVA on the data, which came out significant. However, for the follow-up pairwise, bonferroni adjusted, comparisons, none of them came out significant. (Why this is possible have been covered in several threads before, for example this one.)

Now, I don't really know how to interpret my data. Can I conclude that there at least is a difference between the two levels in my data set with the largest difference, because the ANOVA came out significant, or will I have to reject this conclusion because my pairwise comparisons all came out non-significant?

Note that it might be the case that I, in my specific situation, would be able to find a post-hoc test that came out significant, but my question is more directed towards the general question of how to interpret the analysis results in a situation where a significant post-hoc couldn't be acquired.

Also, this question is somewhat related to mine.

  • $\begingroup$ The Bonferroni adjustment is quite strict. Have you tried to use any other method, for example Holm-Bonferroni? $\endgroup$ – alesc Feb 26 '15 at 21:24
  • $\begingroup$ Well, I'm not that familiar with the post-hoc test world, so I've been using the bonferroni in order to not cross any lines. Plus, SPSS, which I'm using right now, won't let me run any of the other post-hoc tests for this dataset. All the post-hoc options are just greyed out without any explanation. $\endgroup$ – Speldosa Feb 26 '15 at 22:11
  • $\begingroup$ The second test is more stringent than the omnibus F-test that is part of the ANOVA. Therefore, more weight should be put on the second test results. $\endgroup$ – Mike Hunter Oct 31 '15 at 23:31

The reason that the ANOVA rejects could be due to violations in other assumptions such as homogeneity of variances/sphericity. (See assumption #5 here:https://statistics.laerd.com/spss-tutorials/one-way-anova-repeated-measures-using-spss-statistics.php)

If for example sphericity is violated, then you can conclude that the significant result from ANOVA is likely due to violations of the ANOVA assumptions as opposed to a significant difference between two levels.

If the important ANOVA assumptions are met and you still run into this issue, then you can conclude that there is a possible difference between at least two of the levels, but that the significant ANOVA may be a false positive.

This page explains multiple comparisons: http://www.biostathandbook.com/multiplecomparisons.html

Another possible conclusion/explanation for your particular case using Bonferroni adjustments is that the Bonferroni method is very conservative so even if a difference between the levels exist, multiple comparisons with Bonferroni adjustments may not have been able to detect it.


Having a significant omnibus $F$ test is equivalent to saying that there is some contrast among the means, $\sum c_i \bar y_i$ (where $\sum c_i=0$), that is statistically significant when tested using the Scheffe procedure.

In fact, that most-significant contrast has coefficients $c_i = (\bar y_i - \bar y_\cdot)$ (or any nonzero multiple thereof). On the other hand, a pairwise comparison is a contrast with one $c_i=1$ and one $c_i=-1$; so the situation where $F$ is significant but no pairwise comparison is cannot occur when the means all cluster around only two different values. So you must have a more varied distribution of means than that.

One way to avoid the dilemma you describe in routine analysis is to just skip the $F$ test -- or ignore those results in the computer output. It is still a valid statistical procedure to not precede pairwise comparisons with an $F$ test, and if pairwise comparisons are the only contrasts of possible interest, then you lose nothing by skipping the $F$ test. (The exception to this statement is the Fisher protected LSD, which requires the $F$ -- but it does not strongly control the overall error rate, and I don't ever recommend using it.)

Second, you could choose a less conservative method than Bonferroni for the pairwise comparisons. If the assumption of equal variances is tenable and the design is balanced, then the Tukey HSD procedure is "exact" rather than conservative.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.