Testing for answer order bias

Question

Answer order bias is something I occasionally hear about, especially in the context of election ballots. But presumably, it can also occur in various other situations like online polls, multiple-choice exam questions, etc.

Let's say I design a questionnaire system that randomizes the order of answers for users. For any given user, I know their selected choice and the order of the answers that were provided to them. What statistical test(s) would be appropriate for determining if there was answer order bias from this data?

I'm aware there are other types of survey bias, so let's ignore those. Also, assume the answer options are not ordinal (e.g., "low", "med", "high")

Answer 1

This can be determined using a simple one-way design or a regression. Assume that you can create a metric $Y$ which is the measured response to the survey and it has a mean $\mu$. It could be the answer to a specific question, or it could be the percentage of time that a selection is made out of a set. Also assume that $\omega_i$ is a particular ordering in the space of all possible orderings.

One way design

$H_0$: $\mu_{\omega_i} = \mu$

Does the ordering $\omega_i$ create a ordering bias? Estimate $\hat{\mu}_{\omega_i}$ by delivering the survey with the specific order to $n_1$ people and estimate $\hat{\mu}$ by delivering the survey with a random order to $n_2$ people. Use the appropriate 1-way statistical test, depending on the distribution of $\hat{\mu}$ and the number of participants.

Generalized Linear Model

Randomly deliver a possible ordering $\omega_i$ to a large number of people such that each ordering has multiple responses, create a regression on the ordering.

$$g(Y) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_n x_n + \epsilon$$

where $x_i = 1$ if ordering $\omega_i$ was used. Test if the coefficient on ordering $i$ is significantly different from a baseline ordering.