Designing an experiment where participants see all levels of one factor, but not of the other

Question

Let's say I have an experiment with two factors A and B, each with a number of different levels.

I want participants to

• see all levels of factor A exactly $n$ times,
• never see a level of B twice

How would I generate a test plan that assigns me $x$ users to the various combinations of A and B given the above constraints?

More specifically, suppose I have 5 pairs of speakers (factor A) and 10 audio samples (factor B). I would like each speaker to be tested by 12 people. I therefore need to recruit 60 ($5 \times 12$) people to listen to all pairs of speakers. Since the audio samples are so long they couldn't possibly listen to all of them. How do I assign them to the audio samples?

I tried starting with a smaller version.. 3 speakers, 6 audio sources, and 6 listeners (A through F), meaning I get two listeners per speaker. But how can I generalize this?

Perhaps I'm missing the terminology to look it up somewhere, but I only got as far as creating the full test matrix, which isn't really helpful here.

Answer 1

I think the terminology you're looking for is latin squares, graeco-latin squares, and balanced incomplete block designs (BIBDs). You might also look into the literature on choice experiments since I believe they deal with these kinds of problems often.

The design given can be constructed using a graeco-latin square (which is two orthogonal latin squares) for sample 1,2,3 and speaker 1,2,3 and then put a copy of that design for samples 4,5,6.

This approach won't work if you have 5 speakers, 10 samples, and 12 people. You'd need to only use 10 people. One issue of graeco-latin squares is that you have to divide your people into two groups, and each person from one group will have a run with each member of the other group. So in your 5,10 design person A will have runs with F,G,H,I, and J, but never B, C, D, or E. You can try relabeling (A,...,J) to (A,C,E,G,I,B,D,F,H,J) in the second square, or some other strategy to reduce the number of duplicates. There are 4 mutually orthogonal latin squares that are 5x5, see this. From those you can construct 12 different graeco-latin squares by just overlaying two squares, or 4 different squares that have 3 entries at each speaker/sound by overlaying 3 squares, or 1 with 4 entries at each speaker/sound by overlaying all 4 squares.

It might also be worth trying an optimal design algorithm where each person is a block and each block has a number of runs equal to the number of levels of $B$. When a highly regular combinatorial design doesn't exist (both of your examples have |A| prime and |B| an integer multiple of |A| which is nice) often we can get very good performance with optimal designs. The only problem is that sometimes it's very computationally difficult to do this optimization.