0
$\begingroup$

I am comparing the response to a drug under five different conditions. The textbook instruction is to perform a one-way ANOVA followed by a post hoc test if necessary.

The P value from an ANOVA is borderline (0.056), however, my data does not fit the criteria for an ANOVA, as the within-group variation varies greatly between groups. Is it acceptable to perform a post hoc test (in my case, t-tests with a multiple comparison adjustment such as Benjamini-Hockberg would seem appropriate) without doing an ANOVA? If so, why does anyone bother with ANOVAs in the age of computers, when multiple corrected tests can be performed so easily?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Are you dealing with one-way ANOVA, where you woud relate a single factor to an outcome variable? If the variability of the data in each category of your factor is non-constant, that would invalidate both your global ANOVA p-value and your multiplicity-adjusted post-hoc p-values. To trust any of those p-values, you would either want to transform the outcome variable data to stabilize the variance or apply an ANOVA method which can handle non-constancy of variance. See https://pdfs.semanticscholar.org/2d90/7e90e401010d46898b5efd8919ebdec0788c.pdf.

Only if your multiple comparisons were pre-planned could you ignore the global ANOVA p-value (but even then you would want to make sure the data verify the assumptions underlying the ANOVA analysis).

Whether or not you would bother with the global ANOVA p-value therefore depends on whether or not your multiple comparisons were pre-planned. If they were, you could ignore the global ANOVA p-value. If they were not, you couldn't. The age of computers has nothing to do with it, really.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you Isabella for pointing out that the all the p-values would be invalid because of the variation in variance. Regarding your comment on whether the multiple comparisons are pre-planned, the question would then be whether there is any reason not to pre-plan multiple comparisons. Is there ever an advantage to planning an ANOVA into the analysis? Is an analytical strategy including ANOVAs in any way more robust than one without ANOVAs, given that the strategy is fully planned before seeing the data? $\endgroup$
    – D Greenwood
    Commented Sep 17, 2018 at 16:16
  • 1
    $\begingroup$ If you know at the study planning phase - before collecting and seeing the data - what specific comparisons among all possible ones you are interested in, then you can declare those as being pre-planned. Otherwise, you will have to treat them as post-hoc comparisons. In an exploratory study, post-hoc comparisons make more sense. In a confirmatory study, pre-planned comparisons make more sense. How you treat multiple comparisons really depends on the type of study and the nature of the research question(s) being addressed (among other considerations). $\endgroup$
    – Isabella Ghement
    Commented Sep 17, 2018 at 17:59
  • $\begingroup$ FYI, I asked a question about this answer here. Would love to get your take. $\endgroup$
    – Quinn Culver
    Commented May 16 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.