When can I leave counterbalancing factors (between participants) in an ANOVA out of a write up?

Question

The following is a problem that I keep encountering in different forms. For now, I give one example, but I would also be interested in the answers to the more general question.

I have a design with a reaction time based dependent variable $Y$, two hypothesis-relevant between participants factors ($A$ and $B$) and three counterbalancing factor ($C$ [stimuli used in different conditions], $D$ [the hands assigned to different responses], and $E$ [the order of different parts of the reaction time measure]). Except for the DV, none of the variables is continuous. For my research question it is interesting:

1. whether the DV x is overall above zero
2. whether factor $A$ has a significant influence on the DV
3. whether $A$ and $B$ interact in influencing the DV

I perform the analyses with ANOVA. Writing up the results of the whole analysis with all five factors takes up a lot of journal space, so I wonder in which case I can leave out the counterbalancing factors from the analysis:

• Only if there is no effect or interaction whatsoever involving the counterbalancing factors?
• Only if there is no interaction between a counterbalancing factor and one of the hypothesis-relevant effects (e.g., no interaction with $A$, no three-way interaction with $A$ and $B$?
• Does it matter whether an interaction between a counterbalancing factor and a hypothesis-relevant effect is ordinal or disordinal (i.e., whether the effect is "turned around" in one condition)?
• And if I encounter a relevant effect or interaction with a counterbalancing factor, how should I continue: reporting the ANOVA including this factor, or should I use a covariate analysis?
• Would the approach be the same if I had a within-participants design?

Answer 1

You don't leave factors out of an analysis for any of the reasons you state. I'm assuming you're talking about significance when you talk about effect presence and significance of the effect is a poor reason to modify your model. This is a designed study and all of these variables should be orthogonal. Further, you're going to have such small cells in the study given so many variables that high variability of the samples is going to impact significance quite a bit.

You counterbalanced because you expected a main effect of the counterbalancing variable. Therefore it's presence as a main effect isn't terribly meaningful. It's lack of presence isn't usually either. Interactions are the critical thing and you need the main effect in the model in order to look at the interaction.

So interactions are really the only thing worth discussing and it should be discussed in terms of the amount of variability accounted for rather than significance. Perhaps they bring about important issues in terms of methodology for future researchers and perhaps they qualify the meaning of whatever effects are found. Regardless of their significance they should all be reported.

That said, many researchers don't report any of the counterbalancing variables. Perhaps it's well established in a field that they're not directly of interest at all and further it may be that the field relies so much on significance tests that the spurious significance of an individual interaction (remember, they happen 5% of the time by chance) isn't worth examining.

I guess I'm just saying you need to think about your data, your variables, and what things mean. But for goodness sakes don't include or exclude things based on significance. Do keep in mind that those additional factors in the model explain error variance and influence F-ratios.

NOTE: Your concern about whether interactions are ordinal or disordinal expresses a fundamental misunderstanding about interactions. All interactions are cross over. That's not always represented in the observed pattern of means due to main effects (whether significant or not). Consider a 2x2 where the lines cross. They are easily uncrossed by making a main effect separate the lines. Therefore, the suggestion that crossing lines may be a simple diagnostic is an oversimplification.

NOTE: Be careful of the variable designation. For example, did hand matter or handedness? Should it be coded in terms of what's dominant or the side. Furthermore, there are purely technical reasons to counterbalance hand for responding such as the fact that keyboards tend to be consistently faster reporting some keys than others because they scan in a fixed order.