I have trouble imagining realistic research questions that can be answered by a one way ANOVA. Two or more ways with interaction terms and perhaps mixed factors (within and between subjects), sure you will need to do an ANOVA among others, but one way? I speak about one-way ANOVAs with more than two groups. If there are only two groups, that's a straightforward t-test or Wilcoxon variation.

A way to put my question is this: Are there really research questions where all you want to know is whether at least any one of those groups is different from any other group, but where you don't care which ones they are?

Another way to put my question is to state that the heavy lifting is not done by the ANOVA but by the planned contrasts or exhaustive post-hoc tests you do along with it. But do those really rely on the ANOVA? Being variants of t-tests, they only require i.i.d. data from interval variables that needs to be normally distributed unless $n$ is large enough. No ANOVA needed so far. Since you will be doing more than one comparison, you should control your type I error rate. If you're doing all pairwise comparisons and control for the FWER, the consensus seems to be that you don't need the ANOVA (and that you should do a power analysis instead of only taking 30 sample points per group for the CLT). Yet in this frequent scenario, I often see the ANOVA done anyway. Is this just a historic relic? I think this addition can be harmful as it reduces the power of the overall procedure by requiring the ANOVA must be significant before doing the post-hoc tests (discussed in more detail below).

Then there are the planned contrasts where you don't compare all pairs but perhaps only some pre-selected ones and/or some linear combinations of groups (are those two treatments on average better than the average of the other three treatments etc.). It is alleged for example by (Howard Seltman, p.325) that those require to first reject the null hypothesis of the ANOVA:

The same kind of argument applies to looking at your planned comparisons without first “screening” with the overall p-value of the ANOVA. Screening protects your Type 1 experiment-wise error rate, while lack of screening raises it.

The scare-quotes around "screening" seem telling to me. Sure, if you add one more hurdle, your type I error cannot go up for it. All other things being equal, it can at most stay the same (the tests are completely redundant) or decrease (the tests measure something slightly different). But if your only concern was type I error rate, why not just decrease the $\alpha$ level? That seems certainly a cleaner solution than to add some other test that doesn't really address your research question. (I also don't agree with the implicit logic that type I error rate should always be your foremost concern, at least not unless you also do a power analysis.)

Am I overlooking something? Is the condition of the significant ANOVA result perhaps only instated because for pre-selected contrasts, no correction for multiple comparisons is done at all? If yes, how do we know that this is not a case where we should correct for FWER? (Family size would be determined by the number of pre-selected contrasts, if we really select them upfront) How would we know that instead, the prerequisite ANOVA is the correct way to deal with multiple pre-selected contrasts for each of which we leave $\alpha$ at the nominal level?

Edit: And since some of the more complex ANOVA designs can be equivalent to a one-way design with more groups (that cover all the combinations among the two- or more way ANOVA), perhaps my question is even more general and applies to many other ANOVAs. I don't want to overstate my case though. I'm not sure which ones can be equivalent to a one-way design and which ones cannot.

  • 3
    $\begingroup$ This reads like a straw man: "but where you don't care which ones they are" mischaracterizes the purpose of ANOVA (and of statistical analysis generally). ANOVA is used specifically when we do care about individual differences. The archetypical example is where one of the groups is a control and the others represent different treatments or levels of a treatment. The initial ANOVA F-test answers the question "is there any difference worth investigating." When the answer is "likely so," then of course you follow it up by characterizing the differences. $\endgroup$ – whuber Sep 29 '17 at 16:38
  • 1
    $\begingroup$ "The archetypical example is where one of the groups is a control and the others represent different treatments or levels of a treatment." That's still not addressed by the ANOVA itself, but by another variation of post-hoc tests. (I didn't venture an exhaustive list) My question remains broadly the same: why this two step procedure instead of directly tackling your actual research question? $\endgroup$ – David Ernst Sep 29 '17 at 16:42
  • 2
    $\begingroup$ I do not understand the sense in which you see ANOVA as not being direct. If you are implicitly proposing to decompose the "research question" into a set of comparisons--e.g., is treatment 1 better than control, is treatment 2 better than control, etc.--then surely you must include a way to deal with the dependencies among those comparisons. That seems to put you right back where you started. Perhaps the simplest possible solution involves determining at the outset whether it's worth the trouble of making those detailed comparisons at all. $\endgroup$ – whuber Sep 29 '17 at 18:32
  • 2
    $\begingroup$ Already your composite research question "Does any treatment work better than the control?" could not be answered by the ANOVA. There could be one treatment that does slight active harm (not enough to be significantly different from the control) and another that does slightly help (not enough to be significantly different from the control). If those two treatments are significantly different among each other, your ANOVA would reject H0 but you would be mistaken to to conclude that at least one treatment works (even that at least one treatment is just significantly different from control). $\endgroup$ – David Ernst Sep 29 '17 at 18:40
  • 2
    $\begingroup$ This question is framed as being about "ANOVA", but really it's about omnibus tests. You seem to think they are synonymous -- in a comment you refer to the omnibus test as "the ANOVA itself" -- but this is basically like viewing multiple regression as being nothing more than the global F test. $\endgroup$ – Jake Westfall Jun 27 '19 at 23:29

You pose the following question: Are there really research questions where all you want to know is whether at least any one of those groups is different from any other group, but where you don't care which ones they are?

Yes, here is one such example.
Research Question: Do students randomly assigned to different teaching assistants' recitation sections do comparably well on key course assessment indicators (say, the final exam)?

I think the issue with the way you've presented your query is that it seems to suggest only ANOVA with statistically significant results are possible for RQs. However, the RQ here is still reasonable (something someone might want to know), and it so happens that the hope is most likely NOT to find a statistically significant finding.

That said, if your query is specifically, are there other methods than 1-way ANOVA when you are expecting a difference? then I would agree...it might be harder to find an authentic RQ example.

To address the second query posed: Yet in this frequent scenario, I often see the ANOVA done anyway. Is this just a historic relic?

I would argue that starting with the planned comparisons without first confirming that a difference is actually present (i.e., an omnibus test) is a quasi-failure to confirm assumptions of the test. I would argue, thus, that it is not just an historic relic, but a process that should be encouraged (even in the more pedantic examples like an ANOVA). As a reviewer that has encountered more than one manuscript where subsequent significant findings (even with MCP adjustments) were reported when the MANOVA failed to detect a difference...I think there is something to be said in maintaining the omnibus protocol for 1-way ANOVA and subsequent MCPs.

  • $\begingroup$ Gregg H, I feel you are muddling things with your first example, since the OP asked about tests for difference, and there are omnibus tests for equivalence which explicitly provide evidence for equivalence, rather than assuming absence of evidence for difference implies the same thing (instead of, say, low power). See for example, Chapter 7 of Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority (Second). Chapman and Hall/CRC Press. and recent work by Cribbie and Mara. $\endgroup$ – Alexis Nov 27 '19 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.