ANOVA: Non-significant main effect with significant post-hoc test?

Question

I need some help in interpreting a situation in a 4-way ANOVA where a given factor does not show up as significant in the omnibus model, but the post-hoc comparison between categories of that variable shows that the first and the third category differ significantly from each other, whereas neither the first nor the third category siffers significantly from the second (so the effect seems to be additive). How should one report such a model?

The model is for a study assessing HIV-related knowledge of adolescents, and includes a score on a questionnaire as a dependent variable, and gender, age, sexual experience, and relationship status as independent variables. Specifically, age turns up as significant (as expected), gender also seems to be marginally significant (.049, although with a high partial eta-squared) and both sexual experience and relationship status are non-significant. However, sexual experience seems to differ significantly between categories in the post-hoc test. The post-hoc comparisons have been adjusted for familiwise error rate. The model does not include any interaction-terms, as they have been found to be non-significant and removed from the final model.

Thank you for the feedback!

Answer 1

You can report it just that way - the omnibus was not significant, but post-hoc tests were. Although I think orthodoxy is that one wouldn't run the post-hoc tests if the omnibus was not significant unless you had strong reasons to perform the post-hoc test.

In either case, the p-value from an omnibus vs from post-hoc reflect different questions. In the former case, you're asking whether there's systematic variability in the outcome that is associated with the IV. In the latter, you're asking about specific differences between the levels of the IV.