7
$\begingroup$

There has been some discussion about why differences exist between factor significance after an ANOVA and no significance in the pairwise comparisons, and it has been very helpful. However, there is no directions regarding how to proceed. For instance, in my case the overall P value given by a one-way ANOVA for the treatment factor was P=0.0127 but all of the treatments came out as not significantly different from each other following Tukey's LSD. Should I run a different post-hoc test in this case? Which one would you recommend?

Thank you so much!

$\endgroup$
  • $\begingroup$ What does the pattern of data look like? $\endgroup$ – John Feb 1 '13 at 23:46
12
$\begingroup$

This is not an area where there is universal agreement.

My view is that 1) the two tests answer different questions, so it's not surprising that they get different answers. 2) This discrepancy is more a demonstration of the problems of p values and, especially, using cutoff values like p < .05. 3) It also gets at the problems of looking for "cookie cutter" methods of doing statistics.

Elaborating a bit: The overall ANOVA asks about the relationships among all the levels; tests like Tukey's LSD compare individual levels. Those are two different questions. Which one are you interested in? Perhaps both?

You say (in the title) that the results are inconsistent. But you only give evidence that the p-values are on opposite sides of .05. So? Why do you care about that? It would be much better to examine effect sizes. And these effect sizes don't have the problem of being inconsistent, because they don't vary.

Finally, it seems like (but perhaps I am wrong) you are asking for a general solution to a general problem; but really, the questions are particular.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ What do you mean by that the effect sizes don't vary? $\endgroup$ – Speldosa Feb 27 '15 at 2:03
  • $\begingroup$ @Speldosa I meant that the effect size is not related to the sample size. $\endgroup$ – Peter Flom Feb 27 '15 at 14:04
1
$\begingroup$

It's possible that you could have no significant differences among all of the individual means with the most liberal of tests—even a planned comparison. I was just tasking my students to develop simulated data with just this feature.

You do an ANOVA to test the pattern of your data. If the test is passed then, in most situations, all you need to do is then describe that pattern.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

I used SPSS and I used to have the same problem and I have tried different tests in the PostHoc. For equal variance assumed, I suggest you use Dunnet test in which you can have different results if you change the selection in Control category (First or Last) and sometimes in the Test (2-sided, < Control, > Control). For unequal variance assumed, you should use Tamhane's T2. That's my personal experience.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.