# Statistical analysis of competition data

I'm comparing scores for two small groups of individuals that competed in a tournament, and I'm being told that the comparison calls for a Mann-Whitney U test. It feels wrong to me, though: my two sets of scores are fundamentally interdependent because the two groups competed against one another.

Briefly: I have two groups, a control group A (n=10 men) and an experimental group B (n=12 men). Group B received a treatment, and then the members of group A and group B were pitted against each other in a tournament. I'm interested in the degree to which A's vs. B's were successful at beating the opposite-group men they competed against.

In each day of tournament play, two A's and two B's competed. Each game was every man for himself, competing for points (the task is irrelevant, I think). On a single day, there were always 4 competitors - 2 A's and 2 B's. But if any of those individuals reached criterion that day (scored a certain number of points, essentially), he got pulled and replaced the next day with a new player from the same home group (A or B). An individual could reach criterion in up to 7 days (if you went 7 days without reaching criterion you were considered to have lost, and pulled from play). This meant that an individual (say, an A) that played and won in a single day faced only three other competitors - 1 A and 2 B's. But an A that did poorly and stayed for 7 days could face many more players - a bunch of A's and B's - as better competitors cycled through.

So I've given each of the men a rank score that reflects the percentage of opposite-group men that he beat. Let's say an A named Joe competed for two days and faced 1 other A and 3 B's, and he came in second, behind the one other A in the group but above the three B's. His score would be 1.0. If Joe had had a harder time and had taken more days to reach criterion, he would probably have faced more competitors overall, but his score would still have been 1.0 if he had outcompeted all of the B's that he met. The score attempts to measure players' effectiveness at beating opposite-group men, and to allow comparison across men that faced different numbers of competitors.

So the rank scores for the two groups look like this:

A: 1, 1, 1, 0.833333, 0.75, 0.833333, 0.5, 0.333333, 0.333333, 0.5, 0.333333, 0.125

B: 1, 0.5, 0.666667, 0.5, 0.333333, 0.5, 0.5, 0.2, 0.166667, 0

And my question is: is there a more valid way to see if there's a difference between the groups than a Mann-Whitney?

Why not conduct a permutation test? This is a natural situation for it: the null hypothesis of no treatment effect basically says the labels are meaningless. So, keep all the results but permute the labels, always maintaining a group of 10 "controls" and a group of 12 "treatment" subjects. Compute any ranking or relative score between the groups you deem meaningful for every one of the $\binom{22}{10}$ = 646,646 permutations. (You could randomly sample the permutations to save time, but their number is small enough that this brute-force calculation is easily carried out.) That's the permutation distribution for your statistic under the null hypothesis. To determine the p-value, see where the observed value of the statistic falls on the cumulative distribution.