5
$\begingroup$

I'd like to get some advice on one-way ANOVA analysis with Dunnett's T3 post hoc test.

Is it possible to find non-significant result from one-way ANOVA analysis (p=.224), yet find differences in the post hoc test (Dunnett's T3 test between groups results? like p=.039 (p<.005)?

I understand I am supporsed to run a post-hoc test only if I find a statistical difference in the Anova test first. But it was interesting how the post-hoc test reported there were significant differneces among certain groups.

How should I report this? Or should I not report this at all?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

Yes, it is possible for the omnibus ANOVA test statistic (testing the null hypothesis that the data arise from groups with the same mean) to be non-significant, while individual tests (allowing for multiple comparisons) are significant.

This is because the individual tests have greater statistical power to detect a difference than the omnibus test. As such, you can report the results of the individual tests with an explanatory note.

The advice to only run post hoc tests if the omnibus test is significant is due to Fisher, whose Least Significant Difference test requires that the global test null hypothesis be rejected. Modern tests such as Dunnett's are stand-alone.

As a general point I would advise less reliance on p-values and more on effect sizes.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Dear Robert, Thank you so much for your kind and very informative answer. I will go back and calculate the effect sizes. From what I understand, it can be calcualted by sum of square between the groups divided by Total sum of squares. (.01=small effect, .06=medium effect, .14=large effect) Thank you once again for all your help! Super_Mom, Mature Student $\endgroup$
    – Star_Mom
    Jun 17, 2016 at 8:28
  • $\begingroup$ @Star_Mom You're welcome. See here for more information about ANOVA effect sizes. Since you're new so the site, please don't forget to mark the answer as accepted if you are happy with it. See also How does accepting an answer work? $\endgroup$
    – Robert Long
    Jun 17, 2016 at 8:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.