How can I interpret a significant one-way repeated measures ANOVA with non-significant pairwise, bonferroni adjusted, comparisons?

Question

I have a dataset with one factor consisting of four levels. I've run a one-way repeated measures ANOVA on the data, which came out significant. However, for the follow-up pairwise, bonferroni adjusted, comparisons, none of them came out significant. (Why this is possible have been covered in several threads before, for example this one.)

Now, I don't really know how to interpret my data. Can I conclude that there at least is a difference between the two levels in my data set with the largest difference, because the ANOVA came out significant, or will I have to reject this conclusion because my pairwise comparisons all came out non-significant?

Note that it might be the case that I, in my specific situation, would be able to find a post-hoc test that came out significant, but my question is more directed towards the general question of how to interpret the analysis results in a situation where a significant post-hoc couldn't be acquired.

Also, this question is somewhat related to mine.

Answer 1

Having a significant omnibus $F$ test is equivalent to saying that there is some contrast among the means, $\sum c_i \bar y_i$ (where $\sum c_i=0$), that is statistically significant when tested using the Scheffe procedure.

In fact, that most-significant contrast has coefficients $c_i = (\bar y_i - \bar y_\cdot)$ (or any nonzero multiple thereof). On the other hand, a pairwise comparison is a contrast with one $c_i=1$ and one $c_i=-1$; so the situation where $F$ is significant but no pairwise comparison is cannot occur when the means all cluster around only two different values. So you must have a more varied distribution of means than that.

One way to avoid the dilemma you describe in routine analysis is to just skip the $F$ test -- or ignore those results in the computer output. It is still a valid statistical procedure to not precede pairwise comparisons with an $F$ test, and if pairwise comparisons are the only contrasts of possible interest, then you lose nothing by skipping the $F$ test. (The exception to this statement is the Fisher protected LSD, which requires the $F$ -- but it does not strongly control the overall error rate, and I don't ever recommend using it.)

Second, you could choose a less conservative method than Bonferroni for the pairwise comparisons. If the assumption of equal variances is tenable and the design is balanced, then the Tukey HSD procedure is "exact" rather than conservative.

Answer 2

The reason that the ANOVA rejects could be due to violations in other assumptions such as homogeneity of variances/sphericity. (See assumption #5 here:https://statistics.laerd.com/spss-tutorials/one-way-anova-repeated-measures-using-spss-statistics.php)

If for example sphericity is violated, then you can conclude that the significant result from ANOVA is likely due to violations of the ANOVA assumptions as opposed to a significant difference between two levels.

If the important ANOVA assumptions are met and you still run into this issue, then you can conclude that there is a possible difference between at least two of the levels, but that the significant ANOVA may be a false positive.

This page explains multiple comparisons: http://www.biostathandbook.com/multiplecomparisons.html

Another possible conclusion/explanation for your particular case using Bonferroni adjustments is that the Bonferroni method is very conservative so even if a difference between the levels exist, multiple comparisons with Bonferroni adjustments may not have been able to detect it.