Let's say I have an experiment with two factors A and B, each with a number of different levels.

I want participants to

  • see all levels of factor A exactly $n$ times,
  • never see a level of B twice

How would I generate a test plan that assigns me $x$ users to the various combinations of A and B given the above constraints?

More specifically, suppose I have 5 pairs of speakers (factor A) and 10 audio samples (factor B). I would like each speaker to be tested by 12 people. I therefore need to recruit 60 ($5 \times 12$) people to listen to all pairs of speakers. Since the audio samples are so long they couldn't possibly listen to all of them. How do I assign them to the audio samples?

I tried starting with a smaller version.. 3 speakers, 6 audio sources, and 6 listeners (A through F), meaning I get two listeners per speaker. But how can I generalize this?

Perhaps I'm missing the terminology to look it up somewhere, but I only got as far as creating the full test matrix, which isn't really helpful here.


I think the terminology you're looking for is latin squares, graeco-latin squares, and balanced incomplete block designs (BIBDs). You might also look into the literature on choice experiments since I believe they deal with these kinds of problems often.

The design given can be constructed using a graeco-latin square (which is two orthogonal latin squares) for sample 1,2,3 and speaker 1,2,3 and then put a copy of that design for samples 4,5,6.

This approach won't work if you have 5 speakers, 10 samples, and 12 people. You'd need to only use 10 people. One issue of graeco-latin squares is that you have to divide your people into two groups, and each person from one group will have a run with each member of the other group. So in your 5,10 design person A will have runs with F,G,H,I, and J, but never B, C, D, or E. You can try relabeling (A,...,J) to (A,C,E,G,I,B,D,F,H,J) in the second square, or some other strategy to reduce the number of duplicates. There are 4 mutually orthogonal latin squares that are 5x5, see this. From those you can construct 12 different graeco-latin squares by just overlaying two squares, or 4 different squares that have 3 entries at each speaker/sound by overlaying 3 squares, or 1 with 4 entries at each speaker/sound by overlaying all 4 squares.

It might also be worth trying an optimal design algorithm where each person is a block and each block has a number of runs equal to the number of levels of $B$. When a highly regular combinatorial design doesn't exist (both of your examples have |A| prime and |B| an integer multiple of |A| which is nice) often we can get very good performance with optimal designs. The only problem is that sometimes it's very computationally difficult to do this optimization.

  • $\begingroup$ Thank you for your answer, especially the terms. I think the number of people is the variable I can change easily – I just get more testers. Can you maybe refer me to any implementation of such a design algorithm? I think computationally, it doesn't matter how long it takes to find a design… $\endgroup$ – user13907 Apr 13 '14 at 10:04
  • $\begingroup$ When I do optimal design I use JMP or write my own code, and this problem is a little weird because of the constraints, so writing your own code may be necessary (that is to say, I don't know how to do it in JMP). Is 5,10 the real design you want to do? Or are you wanting a general solution? $\endgroup$ – neverKnowsBest Apr 13 '14 at 12:56
  • $\begingroup$ Sorry for the late response. I was rather thinking about a general design, but I've never heard of JMP so I'm going to check that out. $\endgroup$ – user13907 Apr 29 '14 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.