Comparing tournament results to chance

Question

Basically what I want to achieve is to take loads of data from a certain game's tournaments and compare it to chance, i.e., I want to see whether a "coin flipping tournament" would have similar results. Or in other words, whether any given player has ~50% chance to win against any other given player, and at the start all of them have equal chance to win a tournament (due to equal skill level, too much randomness in the game, or other reasons, doesn't matter).

We have players that became champions once, more than once, or never. I assume that would happen in a coin flipping tournament as well. My understanding of statistics is rudimentary at best, so first question is, is this even possible to check whether we have a real skill differences and can predict some players to be more likely to win in the future? Or do we only have pure 50/50 chance? If yes, what would be the method? Or perhaps, what do I need to read up on before starting it?

Answer 1

As I commented, you should not trust the data from the qualifying stage where some games might be meaningless, or one player might have nothing to gain, such as a player with $0$ wins and $2$ losses playing a third game with no chance to win the group. Even if you know that the last games mattered in some group, including that data would introduce a bias. So, concentrate on the games between the top $8$ players.

I don't think it causes a large distortion to ignore the tournament structure and treat all of the games equally. There is a slight bias that players who win get to play more games, which leads to an amusing result that the average winning percentage should be lower than $50\%$ even though the players played each other. See this question in which the same effect arises. To be perfectly safe you could consider just the quarterfinal games, which make up $4/7$ of the data.

If the games were replaced by coin-flips, and you have a lot of data, then the counts of wins for the players would be approximately a multivariate normal distribution. To specify this distribution you need the mean vector and the covariance matrix. If the $i$th player has played $n_i$ games, then the $i$th coordinate of the mean vector is $n_i/2$, and the $i$th diagonal element of the covariance matrix should be $n_i/4$. If the $i$th player has played $n_{ij}$ games against the $j$th player, then the $ij$ and $ji$ entries of the covariance matrix should be $-n_{ij}/4$. Once you have these parameters you can use a multivariate normality test to see whether getting a result as extreme as the observed result is significant.