# Are there realistic/relevant use-cases for one way ANOVA?

I have trouble imagining realistic research questions that can be answered by a one way ANOVA. Two or more ways with interaction terms and perhaps mixed factors (within and between subjects), sure you will need to do an ANOVA among others, but one way? I speak about one-way ANOVAs with more than two groups. If there are only two groups, that's a straightforward t-test or Wilcoxon variation.

A way to put my question is this: Are there really research questions where all you want to know is whether at least any one of those groups is different from any other group, but where you don't care which ones they are?

Another way to put my question is to state that the heavy lifting is not done by the ANOVA but by the planned contrasts or exhaustive post-hoc tests you do along with it. But do those really rely on the ANOVA? Being variants of t-tests, they only require i.i.d. data from interval variables that needs to be normally distributed unless $$n$$ is large enough. No ANOVA needed so far. Since you will be doing more than one comparison, you should control your type I error rate. If you're doing all pairwise comparisons and control for the FWER, the consensus seems to be that you don't need the ANOVA (and that you should do a power analysis instead of only taking 30 sample points per group for the CLT). Yet in this frequent scenario, I often see the ANOVA done anyway. Is this just a historic relic? I think this addition can be harmful as it reduces the power of the overall procedure by requiring the ANOVA must be significant before doing the post-hoc tests (discussed in more detail below).

Then there are the planned contrasts where you don't compare all pairs but perhaps only some pre-selected ones and/or some linear combinations of groups (are those two treatments on average better than the average of the other three treatments etc.). It is alleged for example by (Howard Seltman, p.325) that those require to first reject the null hypothesis of the ANOVA:

The same kind of argument applies to looking at your planned comparisons without first “screening” with the overall p-value of the ANOVA. Screening protects your Type 1 experiment-wise error rate, while lack of screening raises it.

The scare-quotes around "screening" seem telling to me. Sure, if you add one more hurdle, your type I error cannot go up for it. All other things being equal, it can at most stay the same (the tests are completely redundant) or decrease (the tests measure something slightly different). But if your only concern was type I error rate, why not just decrease the $$\alpha$$ level? That seems certainly a cleaner solution than to add some other test that doesn't really address your research question. (I also don't agree with the implicit logic that type I error rate should always be your foremost concern, at least not unless you also do a power analysis.)

Am I overlooking something? Is the condition of the significant ANOVA result perhaps only instated because for pre-selected contrasts, no correction for multiple comparisons is done at all? If yes, how do we know that this is not a case where we should correct for FWER? (Family size would be determined by the number of pre-selected contrasts, if we really select them upfront) How would we know that instead, the prerequisite ANOVA is the correct way to deal with multiple pre-selected contrasts for each of which we leave $$\alpha$$ at the nominal level?

Edit: And since some of the more complex ANOVA designs can be equivalent to a one-way design with more groups (that cover all the combinations among the two- or more way ANOVA), perhaps my question is even more general and applies to many other ANOVAs. I don't want to overstate my case though. I'm not sure which ones can be equivalent to a one-way design and which ones cannot.

• This reads like a straw man: "but where you don't care which ones they are" mischaracterizes the purpose of ANOVA (and of statistical analysis generally). ANOVA is used specifically when we do care about individual differences. The archetypical example is where one of the groups is a control and the others represent different treatments or levels of a treatment. The initial ANOVA F-test answers the question "is there any difference worth investigating." When the answer is "likely so," then of course you follow it up by characterizing the differences.
– whuber
Commented Sep 29, 2017 at 16:38
• "The archetypical example is where one of the groups is a control and the others represent different treatments or levels of a treatment." That's still not addressed by the ANOVA itself, but by another variation of post-hoc tests. (I didn't venture an exhaustive list) My question remains broadly the same: why this two step procedure instead of directly tackling your actual research question? Commented Sep 29, 2017 at 16:42
• I do not understand the sense in which you see ANOVA as not being direct. If you are implicitly proposing to decompose the "research question" into a set of comparisons--e.g., is treatment 1 better than control, is treatment 2 better than control, etc.--then surely you must include a way to deal with the dependencies among those comparisons. That seems to put you right back where you started. Perhaps the simplest possible solution involves determining at the outset whether it's worth the trouble of making those detailed comparisons at all.
– whuber
Commented Sep 29, 2017 at 18:32
• Already your composite research question "Does any treatment work better than the control?" could not be answered by the ANOVA. There could be one treatment that does slight active harm (not enough to be significantly different from the control) and another that does slightly help (not enough to be significantly different from the control). If those two treatments are significantly different among each other, your ANOVA would reject H0 but you would be mistaken to to conclude that at least one treatment works (even that at least one treatment is just significantly different from control). Commented Sep 29, 2017 at 18:40
• This question is framed as being about "ANOVA", but really it's about omnibus tests. You seem to think they are synonymous -- in a comment you refer to the omnibus test as "the ANOVA itself" -- but this is basically like viewing multiple regression as being nothing more than the global F test. Commented Jun 27, 2019 at 23:29

You pose the following question: Are there really research questions where all you want to know is whether at least any one of those groups is different from any other group, but where you don't care which ones they are?

Yes, here is one such example.
Research Question: Do students randomly assigned to different teaching assistants' recitation sections do comparably well on key course assessment indicators (say, the final exam)?

I think the issue with the way you've presented your query is that it seems to suggest only ANOVA with statistically significant results are possible for RQs. However, the RQ here is still reasonable (something someone might want to know), and it so happens that the hope is most likely NOT to find a statistically significant finding.

That said, if your query is specifically, are there other methods than 1-way ANOVA when you are expecting a difference? then I would agree...it might be harder to find an authentic RQ example.

To address the second query posed: Yet in this frequent scenario, I often see the ANOVA done anyway. Is this just a historic relic?

I would argue that starting with the planned comparisons without first confirming that a difference is actually present (i.e., an omnibus test) is a quasi-failure to confirm assumptions of the test. I would argue, thus, that it is not just an historic relic, but a process that should be encouraged (even in the more pedantic examples like an ANOVA). As a reviewer that has encountered more than one manuscript where subsequent significant findings (even with MCP adjustments) were reported when the MANOVA failed to detect a difference...I think there is something to be said in maintaining the omnibus protocol for 1-way ANOVA and subsequent MCPs.

• Gregg H, I feel you are muddling things with your first example, since the OP asked about tests for difference, and there are omnibus tests for equivalence which explicitly provide evidence for equivalence, rather than assuming absence of evidence for difference implies the same thing (instead of, say, low power). See for example, Chapter 7 of Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority (Second). Chapman and Hall/CRC Press. and recent work by Cribbie and Mara. Commented Nov 27, 2019 at 6:14

An interesting historical use-case is RA Fisher's explanation of ANOVA in the chapter 'Intraclass Correlations and the Analysis of Variance' in 'Statistical Methods for Research Workers' (there are several online versions which can be found for instance via the Wikipedia article). There he introduces ANOVA with intraclass correlation as a special case (which is an example of one-way ANOVA).

(An earlier example of ANOVA occured in 1923, which compared yield from different type potatoes and manure, and is an example of two-way ANOVA, but the one-way ANOVA relating to intraclass correlation is just as much a realistic research question / use case. I actually do not see why two-way versus one-way makes a difference in suitability of the application of ANOVA)

The question answered by ANOVA is just this: Where does variation come from? Is it due to intra-class variation or between class variation? One-way, two-way, multi-way. The dimension does not matter for such question. The question answered by ANOVA, in whatever dimension, is just whether samples of the same class correlate or not, whether the class plays a role in the variation.

Another way to put my question is to state that the heavy lifting is not done by the ANOVA but by the planned contrasts or exhaustive post-hoc tests you do along with it. But do those really rely on the ANOVA?

Many research question do not care about specific contrasts. For instance, in Fisher's example the research question could be: does the height of siblings correlate? And you do not care about a specific group of siblings to be different, and look at the overall variance between the groups (is the variance between groups just random variance of individuals, or is there some principle that causes variation between groups?)

Yet in this frequent scenario, I often see the ANOVA done anyway. Is this just a historic relic?

ANOVA can sometimes be done before individual tests of contrasts as a control in the multiple comparisons problem.

Methods which rely on an omnibus test before proceeding to multiple comparisons. Typically these methods require a significant ANOVA, MANOVA, or Tukey's range test. These methods generally provide only "weak" control of Type I error, except for certain numbers of hypotheses.