What test should be used for detecting team imbalances in a game?

I'm playing the iPhone/Steam game Hero Academy. It's a bit like Chess, except there's some randomness, and at the beginning of a game players choose what team they'll use, each with different strengths and weaknesses. Both players can pick the same team.

About 400 of us play in an unofficial league, where we record our games online for Elo rankings. I'm not sure how many are active players, though.

I have the win/loss record for every matchup between the four teams. Draws are very rare. The rarest matchup has 143 games; the most-common matchup has 260. There are 3,234 games in the database.

What is the appropriate test for determining whether the teams are of equal strength?

Let's assume that rock-paper-scissors imbalances are undesirable. That is, if Team A beats Team B which beats Team C which beats Team A, we have an imbalanced game. Every matchup should be fair.

A complication is that many players have preferred teams, so that if Team A is very popular among top players, it'll appear stronger. I suspect we'll have to ignore this effect for now.

Another complication is that there's a slight advantage to going first, despite our efforts to ameliorate this with a handicap. Let's ignore this effect as well.

• Do most players always play with the same team or is there a (considerable) amount of variation in the team selection even for a single ("prototypical") participant? Aug 28, 2012 at 19:06
• When you define imbalance are you speaking about individual games or head-to-head win-loss records? If it is individual games it should be expected to happen occasionally. What does detect imbalance mean? Isn't imbalance something you can simply observe from the database? Aug 28, 2012 at 19:06
• @Michael I mean statistically significant imbalance at, say, 95% confidence. Just because a Team A wins 70 of 150 matches against Team B doesn't necessarily mean we know Team A is disfavored in the matchup -- it could be chance, especially if we only noticed that imbalance when looking at one of the twelve matchups. Aug 28, 2012 at 20:01
• @cardinal Some players play only one team, others play a couple, and some use a randomizer built into the game. I'm not sure of the ratios. Aug 28, 2012 at 20:03
• I don't see how it is relevant that intransitive imbalances are "undesirable." They exist in many games, and you should use a test which doesn't ignore the possibility. Aug 28, 2012 at 20:52

You could try the following test for each pair of options A and B. Your null hypothesis is that the rating formula is accurate and that the games are independent. Compute the number of wins predicted by the rating formula for option A, and compare this with the observed number of wins. If the rating formula predicts that the player using option A will win with probability $p$, add $p$ to the total expected wins, and add $(p(1-p))$ to the total variance according to the null hypothesis. If the games are not overwhelmingly lopsided, then you should be able to use a normal approximation since you have over $100$ data points for each match-up. Determine how extreme the observed result was in terms of standard deviations away from the predicted mean.
Since you would apply this test for each possible match-up, you would expect more false positives if you use a typical significance threshold for a single test. So, instead of asking for the results to be significant at the $0.05$ level on at least one of $6$ tests, you might want to require $0.05/6 \approx 0.008$ or about $2 \frac23$ standard deviations from the mean in either direction in order to reject the null hypothesis.