Advantages of counterbalancing vs. randomizing stimuli

Question

I'm designing an experiment, in which 40 participants answer 10 questions, 5 in condition $A$ and 5 in condition $B$, and I'm interested in the difference between the two conditions.

It's not clear to me what's the best way to allocate questions to each condition. A simple solution is to counterbalance - generate two lists: \begin{align} {\rm List}\ 1\ &=\ Q1A,\ Q2B,\ Q3A,\ Q4B,\ Q5A,\ Q6B,\ Q7A,\ Q8B,\ Q9A,\ Q10B \\ {\rm List}\ 2\ &=\ Q1B,\ Q2A,\ Q3B,\ Q4A,\ Q5B,\ Q6A,\ Q7B,\ Q8A,\ Q9B,\ Q10A \end{align} and randomly assign participants to one or other.

Alternatively, I could randomly assign 5 questions to each condition for each participant, fully randomizing the stimuli.

I plan on analysing the result using a mixed effects model, as I'm expecting some missing cases:

lmer(dv ~ condition + (condition|question) + (condition|subject))

My question is this: Is it more statistically appropriate to counterbalance, or to randomize in such a situation?

Answer 1

I think the pros of counterbalancing are basically their convenience for you. You set up two questionnaires and you're done. If you have many people using each list, you can add List as a factor and test to see if it has any effect.

The cons of counterbalancing are that there may be some effect of, say, $Q1$ and $Q3$ being in the same condition. That is the case now in both of your lists (they are both in $A$ in list 1, and both in $B$ in list 2). In fact there are many such possibilities (all evens in same condition, etc.). There are also possible order effects ($Q1$ always comes before $Q2$, etc.). It is possible to create a set of lists that counterbalances across all possibilities, but that is a lot of permutations. Randomizing makes all possibilities equally likely (in the long run) and thus marginalizes over (washes out) these possible effects. Presumably, these effects are not actually of interest to you (they are nuisance variables). If so, randomizing better controls for these issues. As such, randomization has a theoretical advantage. However, the effects may be quite small in reality; so small in fact, that this isn't something you really need to worry about. And randomization is likely to be a pain.