11
$\begingroup$

I ran ANOVA with dependent variable IQ, independent variable field of study (3 groups: science, humanities, business), and two covariates (age and sex). I see that my result is not quite significant, but very nearly significant (p = .051). Is it still ok to run post hoc comparisons? The reason being, if one or more of the three individual pairwise comparisons is also nearly significant, I would like to report it as a trend towards significance.

Thanks, FBH.

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
10
$\begingroup$

I highly recommend reading Midway et al., 2020, which is probably the best article I have ever read that summarizes pairwise comparisons for ANOVA. Along with the guidelines they provide for how to properly utilize these post-hoc tests, one of the opening paragraphs states this:

The classic ANOVA (ANalysis Of Variance) is a general linear model that has been in use for over 100 years (Fisher, 1918) and is often used when categorical or factor data need to be analyzed. However, an ANOVA will only produce an F -statistic (and associated p-value) for the whole model. In other words, an ANOVA reports whether one or more significant differences among group levels exist, but it does not provide any information about specific group means compared to each other. Additionally, it is possible that group differences exist that ANOVA does not detect. For both of these reasons, a strong and defensible statistical method to compare groups is nearly a requirement for anyone analyzing data.

For this reason, it is useful to explore differences between groups even if there is no significant ANOVA. Remember that the f value derived from an ANOVA simply tests the overall variance between groups. How some groups compare to each other cannot be understood without exploring further.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thanks for your reply. Helps a lot! $\endgroup$
    – FastBallooningHead
    Commented Mar 6, 2023 at 4:55
  • 1
    $\begingroup$ No problem. Feel free to accept one of the answers here (clicking the checkmark next to it) if you feel it answered your question. $\endgroup$
    – Shawn Hemelstrand
    Commented Mar 6, 2023 at 6:34
  • $\begingroup$ Thanks for letting me know. I had been thinking I accomplish this by clicking the up arrow. Will also click the check mark from now on. $\endgroup$
    – FastBallooningHead
    Commented Mar 6, 2023 at 8:54
  • 1
    $\begingroup$ No worries. Good luck with your analysis. $\endgroup$
    – Shawn Hemelstrand
    Commented Mar 6, 2023 at 10:09
13
$\begingroup$

Since multiple comparison tests are often called 'post tests', you'd think they logically follow the one-way ANOVA and should be used only when the overall ANOVA results in $p < 0.05$ (or whatever threshold you choose). In fact, this isn't so.

"An unfortunate common practice is to pursue multiple comparisons only when the null hypothesis of homogeneity is rejected." (1)

With one exception, the results of multiple comparison tests (post-hoc tests) following ANOVA are valid even if the overall ANOVA did not find a statistically significant difference among means. The exception is the first multiple comparison test invented (now obsolete), the protected Fisher Least Significant Difference (LSD) test.

I suggest focusing on confidence intervals of the differences between means, and not on whether any p-value is less than 0.05. And please don't ever use the phrase "trending towards significance". You actually don't know what would happen to the p-value if there were more data, so you can't state a trend (2).

  1. J. Hsu, Multiple Comparisons: Theory and Methods, page 177, ISBN 978-0412982811
  2. Wood J, Freemantle N, King M, Nazareth I (2014) Trap of trends to statistical significance: likelihood of near significant P value becoming more significant with extra data. BMJ Br Medical J 348:g2215. https://doi.org/10.1136/bmj.g2215
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your response. Very helpful! $\endgroup$
    – FastBallooningHead
    Commented Mar 5, 2023 at 17:29
  • 3
    $\begingroup$ +1 for the useful answer. Interestingly, my undergrad stats professor and grad stats professor both disagreed on this point for some bizarre reason, but I think the grad stats professor was correct (who had the same reasoning as you). $\endgroup$
    – Shawn Hemelstrand
    Commented Mar 6, 2023 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.