# How to analyze repeated A/B choice questions?

I feel a bit silly, but I'm stuck on what I thought would be an easy analysis. Haven't had this type of data in a while, so I'm a bit rusty. Tried Googling is, but as I'm not a native English speaker I'm not entirely sure I'm using the right terms.. Anyway, so much for the apology in advance.

Participants in my study answered 24 questions where they simply had to choose between option A and option B. The questions are all "the same" in the sense that H0 is that they will answers 50%/50%; 12 times A and 12 times B. How do I aggregate and analyze?

1. Do I aggregate the data into a percentage A (or B, for that matter) per person and then run a one-sample t-test against 50%?
2. Do I aggregate into counts of A (again, or B of course) and then somehow do a Chi-square? <- this is where it gets fuzzy!
• Well, what question do you want to ask of your data? Commented Nov 22, 2012 at 18:23
• Do people choose A more than B (or B more than A) or is there no difference? Commented Nov 22, 2012 at 18:39
• By the way, there was an answer here as well, which I even replied to, but now it's gone. If I have accidentally deleted it I'm sorry. Commented Nov 22, 2012 at 18:40
• It is unclear what H0 you want to test, either (1) for any single question, the proportion is 50/50; or (2) the sum over all 24 questions is 12 (if to code A=0 and B=1); or (3) some other. Commented Nov 23, 2012 at 7:47
• Not for any single question, as I consider the questions equivalent (also see my reply to Jack Tanner below). I'm interested in whether people have a preference of A over B (or vice versa). In other words, if every individual answers A 12 times and B 12 times there is clearly no preference. Commented Nov 26, 2012 at 8:34

I think your best bet is to use individual as the unit of analysis; so you add up the 24 choices of A for each individual and then you just have a one dimensional dataset equal to the number of subjects in your study. Then you can compare the distribution of these numbers with what you would expect under the null hypothesis (which would be approximately normal with a mean of 0.5 and easily calculated variance, or you could calcuate its exact distribution if you wanted although I personally don't think it's necessary when you have 24 observations each in this sort of domain).

Comparing your actual distribution to the null can be done in several ways, but this should be enough to get you started. I would certainly start graphically eg plotting the empirical density and superimposing the null hypothesis. This would give you a start in seeing if there is a pull towards or away from A, or possibly bi-modal (one group of people prefer A, another group prefer B).

So looking again at your question, my vote is for option 1.

• +1 Peter is exactly right; his answer describes a hypothesis test, and mine describes a confidence interval of the same. And his visualization advice is right on. Commented Nov 28, 2012 at 4:51
• Many thanks. I guess my first intuition wasn't too bad after all. :) Commented Nov 28, 2012 at 13:42

One way to analyze this is a Chi-squared test. Here's an example. But, as you point out, that discards information on repeated measures per individual.

An alternative is a Binomial proportion test. The idea is that you have observed some quantity $a_i$ of A answers from each subject $i$, out of n=24 questions.

$a_i \sim Bin(p,n)$

Here, $p$ is the latent true proportion of preference for A in your sample of subjects. By establishing a confidence interval over $p$, we can see if your expected proportion of 50% A answers falls within the CI; if it does not, you can say that people are significantly more likely to choose A or B.

The R package binom sounds like it's relevant.

• But what goes into the cells? Imagine that I asked people whether they prefer apples or pears 24 times, just phrasing it slightly different every time. I'm not interested in the individual answers on each of the questions (as I consider them equivalent) but whether people prefer apples or pears "overall". However, if I just sum up all the answers and put them in a Chi square table (by summing up I mean add all A's and B's, so if I have 10 participants I could find 130 times A and 110 times B), but am I then not throwing away valueable (per-subject) info? Commented Nov 26, 2012 at 8:31