I'm designing an experiment, in which 40 participants answer 10 questions, 5 in condition $A$ and 5 in condition $B$, and I'm interested in the difference between the two conditions.

It's not clear to me what's the best way to allocate questions to each condition. A simple solution is to counterbalance - generate two lists: \begin{align} {\rm List}\ 1\ &=\ Q1A,\ Q2B,\ Q3A,\ Q4B,\ Q5A,\ Q6B,\ Q7A,\ Q8B,\ Q9A,\ Q10B \\ {\rm List}\ 2\ &=\ Q1B,\ Q2A,\ Q3B,\ Q4A,\ Q5B,\ Q6A,\ Q7B,\ Q8A,\ Q9B,\ Q10A \end{align} and randomly assign participants to one or other.

Alternatively, I could randomly assign 5 questions to each condition for each participant, fully randomizing the stimuli.

I plan on analysing the result using a mixed effects model, as I'm expecting some missing cases:

lmer(dv ~ condition + (condition|question) + (condition|subject))

My question is this: Is it more statistically appropriate to counterbalance, or to randomize in such a situation?


I think the pros of counterbalancing are basically their convenience for you. You set up two questionnaires and you're done. If you have many people using each list, you can add List as a factor and test to see if it has any effect.

The cons of counterbalancing are that there may be some effect of, say, $Q1$ and $Q3$ being in the same condition. That is the case now in both of your lists (they are both in $A$ in list 1, and both in $B$ in list 2). In fact there are many such possibilities (all evens in same condition, etc.). There are also possible order effects ($Q1$ always comes before $Q2$, etc.). It is possible to create a set of lists that counterbalances across all possibilities, but that is a lot of permutations. Randomizing makes all possibilities equally likely (in the long run) and thus marginalizes over (washes out) these possible effects. Presumably, these effects are not actually of interest to you (they are nuisance variables). If so, randomizing better controls for these issues. As such, randomization has a theoretical advantage. However, the effects may be quite small in reality; so small in fact, that this isn't something you really need to worry about. And randomization is likely to be a pain.

  • $\begingroup$ So there's no statistical reason to counterbalance rather than randomize? That's really what I wanted to hear - it's pretty painless to fully randomize condition and order nowadays (thanks, OpenSesame). $\endgroup$ – Eoin Feb 28 '15 at 18:08
  • $\begingroup$ There is no real advantage of counterbalancing in practice, @Eoin. If you did have every permutation represented, w/ adequate nj per permutation, you could test specific hypotheses (assuming you wanted to). But the number of permutations is so large N would be prohibitive, & I assume this is a nuisance. Thus, no there is no reason to counterbalance (rather than fully randomize) other than convenience. $\endgroup$ – gung - Reinstate Monica Feb 28 '15 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.