4
$\begingroup$

I have a 5 x 2 design where one of the levels of the first factor is a control, all others are experimental conditions. I'm interested in the interaction between the two factors and especially, if the interaction is present for each of the 4 experimental conditions with the control condition.

I conduct 4 separate 2 x 2 ANOVAs where I pair each experimental condition with the control condition in the first factor. This seems to call for p-value adjustment to avoid the multiple-testing problem but what do I have to adjust? Are all p-values adjusted (the two main effects and the interaction)? Or do I only adjust the interaction p-values? Or separately for the main effects?

Thanks a lot

$\endgroup$
1
$\begingroup$

If you're just focusing on the four interactions, I would adjust, but only for those 4 tests.

I think most people wouldn't worry about the multiplicity adjustment here; I seldom see such adjustments made unless the number of tests is quite large.

$\endgroup$
4
  • $\begingroup$ I would not mind to adjust but it was required by a reviewer. $\endgroup$
    – thias
    Sep 30 '11 at 12:33
  • 2
    $\begingroup$ @thias - In responding to reviewers' requests on a paper, I'll do whatever they suggest unless I think it's completely wrong, and even then I'll make some modification to the paper in response. $\endgroup$
    – Karl
    Sep 30 '11 at 13:01
  • 2
    $\begingroup$ Good answer, Karl, but could you explain why it's valid to adjust only for the four interaction tests? There also appears to be a complication because the four ANOVAs clearly are dependent: they share the control data. It's unclear then how to adjust the p-values. Shouldn't this testing be done with a single ANOVA, protected by an overall F test, followed by a set of four contrasts to test the interactions? $\endgroup$
    – whuber
    Sep 30 '11 at 16:52
  • 1
    $\begingroup$ @whuber - [Oog; too many important questions.] I'd adjust only for the tests I really cared about (hoping that the reader would trust that I really did only care about those), but of course it's important that what I care about was determined in advance and not after looking at the results. I agree that it's not clear exactly how to adjust, given the dependence. I interpreted his question as that he had done the overall test and was now looking at the individual pieces; it does seem in looking at those pieces that you should account for the fact that there are four of them. $\endgroup$
    – Karl
    Sep 30 '11 at 17:11
1
$\begingroup$

p-value adjustments

P-value adjustments are often designed to control Type I error rates for a set of analyses. There are conventions regarding what is often conceptualised as a set and what is not conceptualised as a set, but such conventions should not be taken too seriously. For example, your four interaction effects of interest might be seen as a set.

Or try to increase parsimony of analyses

Alternatively, you could try to perform your hypothesis testing in a different way in order to minimise the multiplicities in your analysis, or make some analyses conditional on success of previous analyses.

For instance, you could do the following:

  1. First, perform a compound comparison which defines factor 1 as either experimental or control and then tests for the interaction with factor 2. This will tell you in general whether the experimental conditions have a different effect of factor 2 than control.
  2. If the previous compound comparison is significant, you could then do a separate ANOVA (4 x 2) that excludes the control group and thus tests whether there are any differential effects of factor 2 by experimental conditions.
  3. If the previous interaction effect was significant, you could perform some test of which effects in experimental conditions were larger than others (perhaps Tukey's HSD on factor 2 change score).
$\endgroup$
1
  • $\begingroup$ thanks, that seems valid. Actually, that is what I did: I conducted the 5x2 ANOVA first and proceeded to checking the separate 2x2 ANOVAs. However, I also need to report and interprete the main effects of the four ANOVAs. Would I have to adjust across all main effects and interactions together? Or can I have separate sets of adjustments for separate hypotheses (the hypotheses concerning the main effects are largely separate from the interaction hypotheses)...? $\endgroup$
    – thias
    Oct 3 '11 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.