How to analyze repeated A/B choice questions?

Question

I feel a bit silly, but I'm stuck on what I thought would be an easy analysis. Haven't had this type of data in a while, so I'm a bit rusty. Tried Googling is, but as I'm not a native English speaker I'm not entirely sure I'm using the right terms.. Anyway, so much for the apology in advance.

Participants in my study answered 24 questions where they simply had to choose between option A and option B. The questions are all "the same" in the sense that H0 is that they will answers 50%/50%; 12 times A and 12 times B. How do I aggregate and analyze?

1. Do I aggregate the data into a percentage A (or B, for that matter) per person and then run a one-sample t-test against 50%?
2. Do I aggregate into counts of A (again, or B of course) and then somehow do a Chi-square? <- this is where it gets fuzzy!

Answer 1

I think your best bet is to use individual as the unit of analysis; so you add up the 24 choices of A for each individual and then you just have a one dimensional dataset equal to the number of subjects in your study. Then you can compare the distribution of these numbers with what you would expect under the null hypothesis (which would be approximately normal with a mean of 0.5 and easily calculated variance, or you could calcuate its exact distribution if you wanted although I personally don't think it's necessary when you have 24 observations each in this sort of domain).

Comparing your actual distribution to the null can be done in several ways, but this should be enough to get you started. I would certainly start graphically eg plotting the empirical density and superimposing the null hypothesis. This would give you a start in seeing if there is a pull towards or away from A, or possibly bi-modal (one group of people prefer A, another group prefer B).

So looking again at your question, my vote is for option 1.

Answer 2

One way to analyze this is a Chi-squared test. Here's an example. But, as you point out, that discards information on repeated measures per individual.

An alternative is a Binomial proportion test. The idea is that you have observed some quantity $a_i$ of A answers from each subject $i$, out of n=24 questions.

$a_i \sim Bin(p,n)$

Here, $p$ is the latent true proportion of preference for A in your sample of subjects. By establishing a confidence interval over $p$, we can see if your expected proportion of 50% A answers falls within the CI; if it does not, you can say that people are significantly more likely to choose A or B.

The R package binom sounds like it's relevant.