# Comparing tournament results to chance

Basically what I want to achieve is to take loads of data from a certain game's tournaments and compare it to chance, i.e., I want to see whether a "coin flipping tournament" would have similar results. Or in other words, whether any given player has ~50% chance to win against any other given player, and at the start all of them have equal chance to win a tournament (due to equal skill level, too much randomness in the game, or other reasons, doesn't matter).

We have players that became champions once, more than once, or never. I assume that would happen in a coin flipping tournament as well. My understanding of statistics is rudimentary at best, so first question is, is this even possible to check whether we have a real skill differences and can predict some players to be more likely to win in the future? Or do we only have pure 50/50 chance? If yes, what would be the method? Or perhaps, what do I need to read up on before starting it?

-
The type of tournament matters. Many backgammon tournaments are built from 1v1 games, so you can have a lot of data that someone is playing well even if he/she doesn't win the tournament or make it to the money rounds. Most poker tournaments are structured differently, as freezeouts, and the finishing position outside the money is not a reliable indicator of skill. So, what data can you collect? –  Douglas Zare Aug 28 '12 at 6:27
it's 1v1, two group stages of 4 players, two drop two advance in each stage, after that the remaining 8 players play elimination matches. Top8, top4, finals, winner. Group stages have one game, starting with top8 it's best of 7. Top 8 players always get seeded into the next tournament; the rest of participants must fight for the right to play in additional sub-tournament. I oversimplified a bit but I hope that will give you some idea what I am talking about. –  Sejanus Aug 28 '12 at 6:39
I have access to basically all data, who played against whom when, who won, who dropped who advanced, who became the champion, et cetera –  Sejanus Aug 28 '12 at 6:42
Some of the qualifying matches may be tainted because people won't perform optimally if they have nothing to win. However, the results of the final $8$ should all be useful. –  Douglas Zare Aug 28 '12 at 6:52
As I commented, you should not trust the data from the qualifying stage where some games might be meaningless, or one player might have nothing to gain, such as a player with $0$ wins and $2$ losses playing a third game with no chance to win the group. Even if you know that the last games mattered in some group, including that data would introduce a bias. So, concentrate on the games between the top $8$ players.
I don't think it causes a large distortion to ignore the tournament structure and treat all of the games equally. There is a slight bias that players who win get to play more games, which leads to an amusing result that the average winning percentage should be lower than $50\%$ even though the players played each other. See this question in which the same effect arises. To be perfectly safe you could consider just the quarterfinal games, which make up $4/7$ of the data.
If the games were replaced by coin-flips, and you have a lot of data, then the counts of wins for the players would be approximately a multivariate normal distribution. To specify this distribution you need the mean vector and the covariance matrix. If the $i$th player has played $n_i$ games, then the $i$th coordinate of the mean vector is $n_i/2$, and the $i$th diagonal element of the covariance matrix should be $n_i/4$. If the $i$th player has played $n_{ij}$ games against the $j$th player, then the $ij$ and $ji$ entries of the covariance matrix should be $-n_{ij}/4$. Once you have these parameters you can use a multivariate normality test to see whether getting a result as extreme as the observed result is significant.