I'm comparing scores for two small groups of individuals that competed in a tournament, and I'm being told that the comparison calls for a Mann-Whitney U test. It feels wrong to me, though: my two sets of scores are fundamentally interdependent because the two groups competed against one another.

Briefly: I have two groups, a control group A (n=10 men) and an experimental group B (n=12 men). Group B received a treatment, and then the members of group A and group B were pitted against each other in a tournament. I'm interested in the degree to which A's vs. B's were successful at beating the opposite-group men they competed against.

In each day of tournament play, two A's and two B's competed. Each game was every man for himself, competing for points (the task is irrelevant, I think). On a single day, there were always 4 competitors - 2 A's and 2 B's. But if any of those individuals reached criterion that day (scored a certain number of points, essentially), he got pulled and replaced the next day with a new player from the same home group (A or B). An individual could reach criterion in up to 7 days (if you went 7 days without reaching criterion you were considered to have lost, and pulled from play). This meant that an individual (say, an A) that played and won in a single day faced only three other competitors - 1 A and 2 B's. But an A that did poorly and stayed for 7 days could face many more players - a bunch of A's and B's - as better competitors cycled through.

So I've given each of the men a rank score that reflects the percentage of opposite-group men that he beat. Let's say an A named Joe competed for two days and faced 1 other A and 3 B's, and he came in second, behind the one other A in the group but above the three B's. His score would be 1.0. If Joe had had a harder time and had taken more days to reach criterion, he would probably have faced more competitors overall, but his score would still have been 1.0 if he had outcompeted all of the B's that he met. The score attempts to measure players' effectiveness at beating opposite-group men, and to allow comparison across men that faced different numbers of competitors.

So the rank scores for the two groups look like this:

A: 1, 1, 1, 0.833333, 0.75, 0.833333, 0.5, 0.333333, 0.333333, 0.5, 0.333333, 0.125

B: 1, 0.5, 0.666667, 0.5, 0.333333, 0.5, 0.5, 0.2, 0.166667, 0

And my question is: is there a more valid way to see if there's a difference between the groups than a Mann-Whitney?


1 Answer 1


Yes. This situation is so complicated and the results are so interdependent that the applicability of almost any standard test has to be called into question.

Why not conduct a permutation test? This is a natural situation for it: the null hypothesis of no treatment effect basically says the labels are meaningless. So, keep all the results but permute the labels, always maintaining a group of 10 "controls" and a group of 12 "treatment" subjects. Compute any ranking or relative score between the groups you deem meaningful for every one of the $\binom{22}{10}$ = 646,646 permutations. (You could randomly sample the permutations to save time, but their number is small enough that this brute-force calculation is easily carried out.) That's the permutation distribution for your statistic under the null hypothesis. To determine the p-value, see where the observed value of the statistic falls on the cumulative distribution.

BTW, if the men were not chosen to compete at random (formally--not arbitarily--using a random number generator), then one could validly suspect any apparent difference might be due to the sequence in which the men competed. No statistical test can overcome such a deficiency if it is present.

  • 1
    $\begingroup$ I like what @whuber said, but one additional note - if your scores are percentages, they aren't really ranks but ratings. Ranks have integer values, except for ties (which are handled in various ways). $\endgroup$
    – Peter Flom
    Mar 24, 2011 at 10:34
  • $\begingroup$ Really interesting suggestion whuber, and one that feels intuitively correct to me even though I'm not very skilled w/ stats (yet). I will try it (as soon as I find out how, exactly, to run the test). And thanks for the note, Peter Flom. $\endgroup$
    – user3871
    Mar 24, 2011 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.