What test should be used for detecting team imbalances in a game? I'm playing the iPhone/Steam game Hero Academy. It's a bit like Chess, except there's some randomness, and at the beginning of a game players choose what team they'll use, each with different strengths and weaknesses. Both players can pick the same team.
About 400 of us play in an unofficial league, where we record our games online for Elo rankings. I'm not sure how many are active players, though.
I have the win/loss record for every matchup between the four teams. Draws are very rare. The rarest matchup has 143 games; the most-common matchup has 260. There are 3,234 games in the database.
What is the appropriate test for determining whether the teams are of equal strength?
Let's assume that rock-paper-scissors imbalances are undesirable. That is, if Team A beats Team B which beats Team C which beats Team A, we have an imbalanced game. Every matchup should be fair.
A complication is that many players have preferred teams, so that if Team A is very popular among top players, it'll appear stronger. I suspect we'll have to ignore this effect for now.
Another complication is that there's a slight advantage to going first, despite our efforts to ameliorate this with a handicap. Let's ignore this effect as well.
 A: I think it's a bad idea to ignore the strengths of the players, but it may be hard to completely separate the possible flaws in the rating system from the possible advantages of one option versus another. 
You could try the following test for each pair of options A and B. Your null hypothesis is that the rating formula is accurate and that the games are independent. Compute the number of wins predicted by the rating formula for option A, and compare this with the observed number of wins. If the rating formula predicts that the player using option A will win with probability $p$, add $p$ to the total expected wins, and add $(p(1-p))$ to the total variance according to the null hypothesis. If the games are not overwhelmingly lopsided, then you should be able to use a normal approximation since you have over $100$ data points for each match-up. Determine how extreme the observed result was in terms of standard deviations away from the predicted mean.
Since you would apply this test for each possible match-up, you would expect more false positives if you use a typical significance threshold for a single test. So, instead of asking for the results to be significant at the $0.05$ level on at least one of $6$ tests, you might want to require $0.05/6 \approx 0.008$ or about $2 \frac23$ standard deviations from the mean in either direction in order to reject the null hypothesis.
If you reject the null hypothesis, it doesn't necessarily mean that team A has an advantage over team B. It could also be that the ratings formula fails, which might happen for lopsided matches. If you have enough data you can try to compare players of similar ratings, where you can expect the ratings formula to be more accurate. 
