ANOVA - False negatives due to many nonsignificant treatments where one treatment is significant

Question

I'm currently struggling with the following problem and would like to ask you what in your opinion is the best way to deal with this problem (and maybe can recommend literature about the same topic):

I have conducted an experiment where I have exposed 25 individuals in randomized order to 8 treatments each (1 control + 7 treatments). Post-hoc tests (as well as a previous experiment) suggest that only one of the treatment induces a behavioral change that differs significantly from the control treatment (p = 0.01), all other p > 0.2.

However, when I run an anova over all 8 treatments so as to test for the effect of "treatment" , the result is nonsignificant (p = 0.11). This seems to be a false negative although I need a significant value here to justify conducting those post-hoc analyses.

The statistical method I currently apply are LMEs (R package "nlme") (data is normally distributed) with individual ID as random effect.

Do you know how to solve this problem of a false negative aside from increasing sample size?

Thanks for your help!

Answer 1

If you planned a priori to compare each treatment to the control then there is nothing post hoc about the comparisons. The ANOVA does not have to be significant to do the comparisons. In fact doing the ANOVA is not a good idea. However, you should do something to control the Type I error rate. Dunnett's test is probably the best bet since the Bonferroni correction is so conservative.

Answer 2

Why ANOVA before post hoc is not really necessary:

http://stats.stackexchange.comquestions9751do-we-need-a-global-test-before-post-hoc-tests.

Maxwell and Delaney (2004) ...these methods [e.g., Bonferroni, Tukey, Dunnet, etc.] should be viewed as substitutes for the omnibus test because they control alphaEW at three desired level all by themselves. Requiring a significant omnibus test before proceeding to perform any of these analyses, as is sometimes done, only serves to lower alphaEW below the desired level (Bernhardson, 1975) and hence inappropriately decreases power (p. 236)

It is not uncommon to find what appears to be a conflict between the results of the one-way ANOVA and a post hoc test such as Tukey's post hoc test where one finds a statistically significant result for one, but not the other. For example, a statistically significant one-way ANOVA, but no pairwise comparison using the Tukey method that is statistically significant. There can be different reasons for this, such as the conservative or liberal nature of a particular test, but fundamentally it is due to the differences in the distributions used in the one-way ANOVA and Tukey post hoc test (Hsu, 1996). Alternately, you can have a statistically significant Tukey post hoc test, but a non-significant one-way ANOVA. Whether the conclusions from both these tests are in agreement depends on the distribution of the means (Kirk, 2013). In this case, I trust the post hoc analysis instead of the omnibus ANOVA. Thus whatever ANOVA tells me I still go to posthoc analysis to determine significance.

Besides, reporting main effect (like Genotype) is misleading and is just a matter of common sense (Howell, 2010).

Usually, one-way ANOVA's corresponding post hoc test is Tukey's test (if variances of each group are equal) or Games-Howell test (if variances of each group are NOT equal). The reason of not using multiple t-tests for each pair of groups is to control false positive rate. Personally, I use oneway.test for ANOVA and userfriendlyscience::posthocTGH for post hoc analysis in R.

Besides, given your small sample size, it is not surprising the ANOVA returns a not significant result.