4
$\begingroup$

I have an experiment where I measure the effect of a manipulation on two different groups of participants, and the variable that I manipulate also has two levels. Thus, I can essentially use a 2x2 ANOVA to analyze the results. But it is also possible for me to measure the effect of the different levels of the variable first on Group 1 and then on Group 2, by using paired-samples t-tests.

Is there a rule of thumb to decide what needs to be done in this case? I think one advantage of using an ANOVA could be the option to investigate the interaction between the different groups and the different levels of the variable. Perhaps one could also argue that an omnibus analysis is also a better choice from a stylistic point of view.

But is there a very compelling conceptual reason to prefer a 2x2 ANOVA over two separate pairwise comparisons in my case? And if so, what would it be?

Improve this question
$\endgroup$
7
  • $\begingroup$ If there are two groups of participants, and the manipulated variable has two levels, how does the outcome variable look? $\endgroup$
    – IWS
    Commented Nov 1, 2017 at 9:20
  • $\begingroup$ My test measures how young children (preschoolers) and older children (elementary school students) interpret some sentences when they are presented with different word orders. To put it very crudely, it appears that older children do well in both sentence types, but younger children are better in basic word order. So I expect an ANOVA would show a main effect of word order for the younger group, and also an age-word order interaction, because only the younger group seems to have problems with one of the word orders. $\endgroup$
    – Freya
    Commented Nov 1, 2017 at 9:28
  • $\begingroup$ And how do you measure how well these sentences are interpreted? $\endgroup$
    – IWS
    Commented Nov 1, 2017 at 9:30
  • $\begingroup$ Behavioral responses. Kids are supposed to manipulate certain toys based on what they heard. If they are confused about what is the subject and what is the object, they can reverse the event they are supposed to act out with the toys. $\endgroup$
    – Freya
    Commented Nov 1, 2017 at 9:32
  • $\begingroup$ Ok, and how does this outcome data look exactly: do you have some kind of score or likert scale, or is the manipulation rated good vs bad? (I ask because an ANOVA is only applicable if the outcome variable is continuous, which a likert scale may not be, and a 'good vs bad' rating is not) $\endgroup$
    – IWS
    Commented Nov 1, 2017 at 9:34

1 Answer 1

Reset to default
4
$\begingroup$

You Don't Have To Use ANOVA

If your sole objective is confirmation, using pairwise $t$-tests with appropriate multiple testing correction is not wrong. However, there are several reasons why it is preferable to use a model generally speaking, such as ANOVA.

A Model is More Informative

Performing multiple comparisons does not tell you how well these categorical variables explain the variation in the response variable. Nor does it allow you to assess potential interactions, as you pointed out yourself. A $t$-test is just that: a test. Its use is confirmation, while a model can also be used for further inference or prediction (not saying ANOVA is particularly suited for prediction). There is substantial criticism on the overuse of null-hypothesis significance testing. Not every research question is answered by a $p$-value or a confidence interval that doesn't include zero, nor should it be. Again, tests are only for confirmation.

A Model is Easier to Diagnose

Performing diagnostics on each of the individual comparisons is both cumbersome and more prone to false positives for e.g. violation of normality. Having a single ANOVA model allows you to instead inspect one set of residuals to assess the validity of the entire model.

Multiple Comparisons Need to be Corrected For

If you are unfamiliar with multiple testing issues, see this xkcd for example. In my opinion, the most important reason to refrain from using pairwise $t$-tests is that individual comparison's from Tukey's HSD are corrected for multiple testing by default. A newcomer to statistics trying to perform pairwise $t$-tests might end up simply making multiple calls to t.test() without regard for the inflated false positive rate from performing multiple hypothesis tests.

Sometimes T-Tests are Preferred

Say you are only interested in a subset of all possible comparisons. In this case, Tukey's HSD would be a huge waste of power, because a correction will be applied for many tests that didn't interest you in the first place.

Then Again, Some of Those Times Models are Preferred

In the special case where you are comparing everything to a control group, you might also want to make use of lm() instead of aov(), with the control group in the intercept. This has both the advantage of preserving power and the advantages of using a model instead of a test.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ If you compare everything to a control group, you will still need the post-hoc t-tests. Firstly because you will want to know which group was different from the control, which an anova can't tell you. Secondly because an anova can give you false positives that t-tests wouldn't (the anova is significant if no treatment is significantly different from control but two treatments are different from each other which can happen if both are close to no effect but one with a tendency to be harmful). General NHST critique either applies to both anovas and t-tests or to none. $\endgroup$
    – David Ernst
    Commented Nov 2, 2017 at 10:19
  • $\begingroup$ That doesn't take away from the fact that comparing everything to everything (ANOVA followed by Tukey's HSD) involves more $t$-tests than comparing everything to a control group (looking at the $p$-values of a linear model). My point was that if you are not interested in all pairwise comparisons, the summary of a linear model can prevent unnecessary hypothesis testing in the case described. $\endgroup$
    – Frans Rodenburg
    Commented Nov 2, 2017 at 10:30
  • 1
    $\begingroup$ No disagreement that you should only test those that can be potentially interesting, but "My point was that if you are not interested in all pairwise comparisons, the summary of a linear model can prevent unnecessary hypothesis testing in the case described." is either false (the linear model doesn't provide that information) or a semantic trick (the linear model still doesn't provide that information but the t-tests the software launches together with it can. You could just launch those t-tests on their own instead of ANOVA. Regression with some continuous predictors is something else though) $\endgroup$
    – David Ernst
    Commented Nov 2, 2017 at 10:46
  • $\begingroup$ Actually that's not entirely true, first and I think most important: most statistical software includes $p$-values in the summary of a linear model. A beginner in analysis will think these are part of the linear model. Second, an ANOVA strictly speaking doesn't involve t-tests. An omnibus F-test is performed. The post hoc, Tukey's HSD is a series of studentized range test, which admittedly is essentially a $t$-test. $\endgroup$
    – Frans Rodenburg
    Commented Nov 2, 2017 at 10:49
  • 1
    $\begingroup$ For the record, I agree that potential interaction effects require a linear model (as would mixed categorical and continuous variables). Now "most statistical software includes pp-values in the summary of a linear model. A beginner in analysis will think these are part of the linear model" is certainly true, but isn't that precisely the problem? It will lead to a ritual of "just always click the anova button" which triggers a whole battery of tests (Levene + Jarque-Bera + F-tests + exhaustive post-hoc t-tests) when often a few Welch t-tests with correction would have done the trick. $\endgroup$
    – David Ernst
    Commented Nov 2, 2017 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.