0
$\begingroup$

I am aware that there are many questions on this site with basically the exact same title. I have read them all and still want to formulate my question as follows:

I am trying to understand the benefits of running an ANOVA followed by post-hoc t-tests, vs. just running a sequence of t-tests to start with.

I have data for a study where there are three treatment groups for people receiving different kinds of psychotherapy, with three different measurement periods (pre-treatment, post-treatment, six-month follow-up). (The measure is a number resulting from a depression questionnaire.)

My understanding is I can set this up as a repeated measures ANOVA. If I run that, and see a significant result (p < some pre-determined value), my understanding is that all I know at that point is that there was some significant change across one or more time intervals for one or more of the treatment groups, and to find out which time/treatment combination(s) actually showed a significant difference I have to run a post-hoc t-test for each combination -- and to counteract the increased probability of getting a type I (false positive) error by running so many t-tests I can adjust the p-value limit to make it more strict.

If all of that is right, I'm having trouble understanding why (at least in my particular 3x3 case) it's better to use the ANOVA at all, instead of just doing the multiple t-tests with stricter p-values in the first place.

One thing I can understand is that if the result of the ANOVA is not significant, it's easier to find that out with the ANOVA instead of having to do the multiple t-tests. But is that the only reason? Or is the ANOVA doing something more for me?

$\endgroup$

1 Answer 1

1
$\begingroup$

There are a lot of paired comparisons among 3 treatments and 3 time periods. Depending on your level of curiosity, perhaps as many as ${9\choose 2}= 36$ comparisons altogether.

Suppose there are no true differences anywhere in the experimental design, then each paired test at the usual 5% significance level has probability $0.05$ of rejecting $H_0$ to make a 'false discovery'. At worst, that might mean

  • An average of $36(.05) = 1.8$ false discoveries per experiment and
  • A probability $1 - (.95)^{36} \approx 0.84$ of at least one false discovery.

The advantage of doing an ANOVA first is that that you can do one test at the 5% level to see whether there are any differences at all. Then if you reject the overall $H_0$ at the 5% level, you can take an approach towards finding true differences that keeps the probability of false discovery low.

$\endgroup$
3
  • $\begingroup$ Your answer seems to confirm that what I said in the last paragraph of the OP is correct, that the ANOVA serves as a screening without sacrificing type I error karma. But still, my question is: Suppose the ANOVA shows there is some significant difference somewhere. Then we have to do the repeated t-tests to find where, and we have to employ some approach (as you say) to keep the probability of a false positives low. But if we have such an approach, why not just use it in the first place and do the t-tests and skip the ANOVA? It's maybe a bit more work, but it seems conceptually simpler. $\endgroup$
    – M Katz
    Dec 7, 2020 at 22:43
  • $\begingroup$ One crucial difference, as you say; If ANOVA finds nothing, then you're done. No ad hoc tests. No worries about false discovery. // Also the ANOVA has the best power for legitimate detection of any real effects. Best practice is to start with ANOVA, look at residuals for departure from assumption, etc. $\endgroup$
    – BruceET
    Dec 7, 2020 at 23:02
  • $\begingroup$ Thanks for the clarification. $\endgroup$
    – M Katz
    Dec 8, 2020 at 3:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.