ANOVA results do not match post-hoc Tukey test, how to proceed?

Question

There has been some discussion about why differences exist between factor significance after an ANOVA and no significance in the pairwise comparisons, and it has been very helpful. However, there is no directions regarding how to proceed. For instance, in my case the overall P value given by a one-way ANOVA for the treatment factor was P=0.0127 but all of the treatments came out as not significantly different from each other following Tukey's LSD. Should I run a different post-hoc test in this case? Which one would you recommend?

Thank you so much!

Answer 1

This is not an area where there is universal agreement.

My view is that 1) the two tests answer different questions, so it's not surprising that they get different answers. 2) This discrepancy is more a demonstration of the problems of p values and, especially, using cutoff values like p < .05. 3) It also gets at the problems of looking for "cookie cutter" methods of doing statistics.

Elaborating a bit: The overall ANOVA asks about the relationships among all the levels; tests like Tukey's LSD compare individual levels. Those are two different questions. Which one are you interested in? Perhaps both?

You say (in the title) that the results are inconsistent. But you only give evidence that the p-values are on opposite sides of .05. So? Why do you care about that? It would be much better to examine effect sizes. And these effect sizes don't have the problem of being inconsistent, because they don't vary.

Finally, it seems like (but perhaps I am wrong) you are asking for a general solution to a general problem; but really, the questions are particular.

Answer 2

It's possible that you could have no significant differences among all of the individual means with the most liberal of tests—even a planned comparison. I was just tasking my students to develop simulated data with just this feature.

You do an ANOVA to test the pattern of your data. If the test is passed then, in most situations, all you need to do is then describe that pattern.

Answer 3

I used SPSS and I used to have the same problem and I have tried different tests in the PostHoc. For equal variance assumed, I suggest you use Dunnet test in which you can have different results if you change the selection in Control category (First or Last) and sometimes in the Test (2-sided, < Control, > Control). For unequal variance assumed, you should use Tamhane's T2. That's my personal experience.