I am comparing the response to a drug under five different conditions. The textbook instruction is to perform a one-way ANOVA followed by a post hoc test if necessary.

The P value from an ANOVA is borderline (0.056), however, my data does not fit the criteria for an ANOVA, as the within-group variation varies greatly between groups. Is it acceptable to perform a post hoc test (in my case, t-tests with a multiple comparison adjustment such as Benjamini-Hockberg would seem appropriate) without doing an ANOVA? If so, why does anyone bother with ANOVAs in the age of computers, when multiple corrected tests can be performed so easily?


Are you dealing with one-way ANOVA, where you woud relate a single factor to an outcome variable? If the variability of the data in each category of your factor is non-constant, that would invalidate both your global ANOVA p-value and your multiplicity-adjusted post-hoc p-values. To trust any of those p-values, you would either want to transform the outcome variable data to stabilize the variance or apply an ANOVA method which can handle non-constancy of variance. See https://pdfs.semanticscholar.org/2d90/7e90e401010d46898b5efd8919ebdec0788c.pdf.

Only if your multiple comparisons were pre-planned could you ignore the global ANOVA p-value (but even then you would want to make sure the data verify the assumptions underlying the ANOVA analysis).

Whether or not you would bother with the global ANOVA p-value therefore depends on whether or not your multiple comparisons were pre-planned. If they were, you could ignore the global ANOVA p-value. If they were not, you couldn't. The age of computers has nothing to do with it, really.

| cite | improve this answer | |
  • $\begingroup$ Thank you Isabella for pointing out that the all the p-values would be invalid because of the variation in variance. Regarding your comment on whether the multiple comparisons are pre-planned, the question would then be whether there is any reason not to pre-plan multiple comparisons. Is there ever an advantage to planning an ANOVA into the analysis? Is an analytical strategy including ANOVAs in any way more robust than one without ANOVAs, given that the strategy is fully planned before seeing the data? $\endgroup$ – D Greenwood Sep 17 '18 at 16:16
  • $\begingroup$ If you know at the study planning phase - before collecting and seeing the data - what specific comparisons among all possible ones you are interested in, then you can declare those as being pre-planned. Otherwise, you will have to treat them as post-hoc comparisons. In an exploratory study, post-hoc comparisons make more sense. In a confirmatory study, pre-planned comparisons make more sense. How you treat multiple comparisons really depends on the type of study and the nature of the research question(s) being addressed (among other considerations). $\endgroup$ – Isabella Ghement Sep 17 '18 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.