How to analyze two-way within subjects design with binary data

Question

I am interested in testing how users perform on three different user interfaces. On each of the three interfaces, a person will try 12 Tasks. So each participant will do Task 1 three times, Task 2 three times, etc. So, in essence the experiment looks like a 3 x 12 (Interface x Task) within subjects design (where each person does 36 tasks). The sample size will be on the small side (roughly a dozen participants).

I am wondering how to analyze completion data, which is binary. If a participant is successful completing a task, then they get a 1. For failure, they score a 0. Ideally, I would like an analysis that is something like an ANOVA so I could see if there are differences in Interfaces, Tasks, or an interaction. If I had to pick though, I am most interested in differences between Interfaces.

Answer 1

If I understand you, you are describing data appropriate to Cochran's Q test (i.e. repeated measures, binary outcomes), but with the additional wrinkle of multiple tests per treatment per subject. I wonder if Berry and Mielke's (2003) Longitudinal Analysis of Data With Multiple Binary Category Choices. Psychological Reports. 93:127–131 might be of use?

Summary—In many experiments, subjects mark all categories that apply when responding to a cafeteria or multiple-response question. One exact and two approximate permutation methods are described to analyze binary answers to multiple-response questions in longitudinal experimental designs, wherein the same or matched subjects respond to the same multiple-response question over two or more trials. The described methods provide probabilities, under the null hypothesis, that the multiple binary responses do not differ over trials.

Also, with respect to sample size: you have $n=12\times3\times12=312$, which is, perhaps, a bit larger than you realize.