The following is a problem that I keep encountering in different forms. For now, I give one example, but I would also be interested in the answers to the more general question.
I have a design with a reaction time based dependent variable $Y$, two hypothesis-relevant between participants factors ($A$ and $B$) and three counterbalancing factor ($C$ [stimuli used in different conditions], $D$ [the hands assigned to different responses], and $E$ [the order of different parts of the reaction time measure]). Except for the DV, none of the variables is continuous. For my research question it is interesting:
- whether the DV x is overall above zero
- whether factor $A$ has a significant influence on the DV
- whether $A$ and $B$ interact in influencing the DV
I perform the analyses with ANOVA. Writing up the results of the whole analysis with all five factors takes up a lot of journal space, so I wonder in which case I can leave out the counterbalancing factors from the analysis:
- Only if there is no effect or interaction whatsoever involving the counterbalancing factors?
- Only if there is no interaction between a counterbalancing factor and one of the hypothesis-relevant effects (e.g., no interaction with $A$, no three-way interaction with $A$ and $B$?
- Does it matter whether an interaction between a counterbalancing factor and a hypothesis-relevant effect is ordinal or disordinal (i.e., whether the effect is "turned around" in one condition)?
- And if I encounter a relevant effect or interaction with a counterbalancing factor, how should I continue: reporting the ANOVA including this factor, or should I use a covariate analysis?
- Would the approach be the same if I had a within-participants design?
