I'm playing the iPhone/Steam game Hero Academy. It's a bit like Chess, except there's some randomness, and at the beginning of a game players choose what team they'll use, each with different strengths and weaknesses. Both players can pick the same team.
About 400 of us play in an unofficial league, where we record our games online for Elo rankings. I'm not sure how many are active players, though.
I have the win/loss record for every matchup between the four teams. Draws are very rare. The rarest matchup has 143 games; the most-common matchup has 260. There are 3,234 games in the database.
What is the appropriate test for determining whether the teams are of equal strength?
Let's assume that rock-paper-scissors imbalances are undesirable. That is, if Team A beats Team B which beats Team C which beats Team A, we have an imbalanced game. Every matchup should be fair.
A complication is that many players have preferred teams, so that if Team A is very popular among top players, it'll appear stronger. I suspect we'll have to ignore this effect for now.
Another complication is that there's a slight advantage to going first, despite our efforts to ameliorate this with a handicap. Let's ignore this effect as well.