Do I need p-value adjustment for multiple ANOVAs on the same dataset? I have a 5 x 2 design where one of the levels of the first factor is a control, all others are experimental conditions. I'm interested in the interaction between the two factors and especially, if the interaction is present for each of the 4 experimental conditions with the control condition.
I conduct 4 separate 2 x 2 ANOVAs where I pair each experimental condition with the control condition in the first factor. This seems to call for p-value adjustment to avoid the multiple-testing problem but what do I have to adjust? Are all p-values adjusted (the two main effects and the interaction)? Or do I only adjust the interaction p-values? Or separately for the main effects?
Thanks a lot
 A: If you're just focusing on the four interactions, I would adjust, but only for those 4 tests.  
I think most people wouldn't worry about the multiplicity adjustment here; I seldom see such adjustments made unless the number of tests is quite large.
A: p-value adjustments
P-value adjustments are often designed to control Type I error rates for a set of analyses. There are conventions regarding what is often conceptualised as a set and what is not conceptualised as a set, but such conventions should not be taken too seriously. For example, your four interaction effects of interest might be seen as a set.
Or try to increase parsimony of analyses
Alternatively, you could try to perform your hypothesis testing in a different way in order to minimise the multiplicities in your analysis, or make some analyses conditional on success of previous analyses.
For instance, you could do the following:


*

*First, perform a compound comparison which  defines factor 1 as either experimental or control and then tests for the interaction with factor 2. This will tell you in general whether the experimental conditions have a different effect of factor 2 than control.

*If the previous compound comparison is significant, you could then do a separate ANOVA (4 x 2) that excludes the control group and thus tests whether there are any differential effects of factor 2 by experimental conditions.

*If the previous interaction effect was significant, you could perform some test of which effects in experimental conditions were larger than others (perhaps Tukey's HSD on factor 2 change score). 

