# How can I interpret a significant one-way repeated measures ANOVA with non-significant pairwise, bonferroni adjusted, comparisons?

I have a dataset with one factor consisting of four levels. I've run a one-way repeated measures ANOVA on the data, which came out significant. However, for the follow-up pairwise, bonferroni adjusted, comparisons, none of them came out significant. (Why this is possible have been covered in several threads before, for example this one.)

Now, I don't really know how to interpret my data. Can I conclude that there at least is a difference between the two levels in my data set with the largest difference, because the ANOVA came out significant, or will I have to reject this conclusion because my pairwise comparisons all came out non-significant?

Note that it might be the case that I, in my specific situation, would be able to find a post-hoc test that came out significant, but my question is more directed towards the general question of how to interpret the analysis results in a situation where a significant post-hoc couldn't be acquired.

Also, this question is somewhat related to mine.

• The Bonferroni adjustment is quite strict. Have you tried to use any other method, for example Holm-Bonferroni? Feb 26, 2015 at 21:24
• Well, I'm not that familiar with the post-hoc test world, so I've been using the bonferroni in order to not cross any lines. Plus, SPSS, which I'm using right now, won't let me run any of the other post-hoc tests for this dataset. All the post-hoc options are just greyed out without any explanation. Feb 26, 2015 at 22:11
• The second test is more stringent than the omnibus F-test that is part of the ANOVA. Therefore, more weight should be put on the second test results. Oct 31, 2015 at 23:31

The reason that the ANOVA rejects could be due to violations in other assumptions such as homogeneity of variances/sphericity. (See assumption #5 here:https://statistics.laerd.com/spss-tutorials/one-way-anova-repeated-measures-using-spss-statistics.php)

If for example sphericity is violated, then you can conclude that the significant result from ANOVA is likely due to violations of the ANOVA assumptions as opposed to a significant difference between two levels.

If the important ANOVA assumptions are met and you still run into this issue, then you can conclude that there is a possible difference between at least two of the levels, but that the significant ANOVA may be a false positive.

Having a significant omnibus $F$ test is equivalent to saying that there is some contrast among the means, $\sum c_i \bar y_i$ (where $\sum c_i=0$), that is statistically significant when tested using the Scheffe procedure.
In fact, that most-significant contrast has coefficients $c_i = (\bar y_i - \bar y_\cdot)$ (or any nonzero multiple thereof). On the other hand, a pairwise comparison is a contrast with one $c_i=1$ and one $c_i=-1$; so the situation where $F$ is significant but no pairwise comparison is cannot occur when the means all cluster around only two different values. So you must have a more varied distribution of means than that.
One way to avoid the dilemma you describe in routine analysis is to just skip the $F$ test -- or ignore those results in the computer output. It is still a valid statistical procedure to not precede pairwise comparisons with an $F$ test, and if pairwise comparisons are the only contrasts of possible interest, then you lose nothing by skipping the $F$ test. (The exception to this statement is the Fisher protected LSD, which requires the $F$ -- but it does not strongly control the overall error rate, and I don't ever recommend using it.)
• You might also be interested in stats.stackexchange.com/questions/83131/… -- where it is shown that the un-corrected P value can be as high as $.15+$ when the $F$'s $P~$value is $.05$. Nov 2, 2015 at 16:44