# How do you verify a rating system?

Let's say I have a database of a lot of (thousands) chess games. In chess, every player have a rating, called Elo that gives an estimation how strong is this player. If we know the the Elo ranking of both players, we can give an estimation of the probabilities of the possible outcomes of the match. In theory, a player whose rating is 100 points greater than their opponent's is expected to score 64%.

This is theory. The player's true skill can be different. E.g. if he/she is really tired, or out of practice, his/her true skill will be lower than his/her Elo. So for every game I have an estimation (if we don't count the draw, it's pratcically a simple binomial experiment) of the probability of the different outcomes, and the outcome itself. Is there a statistic method, with which I can verify how accurate is this estimation?

I need something that can tell how accurate is the method of estimation not just disprove a null hypotesis that it doesn't have anything to do with player's true skills.

• Perhaps you can clarify a bit: (a) What is the relationship between players' ratings and expected winning of a match? (b) How much data do you have? – Joel W. Jun 19 '14 at 13:32
• (a) The higher the rating the more chance of winning. However, it's not the rating that counts. It's the estimated probability of winning. ELO rating is just an example which gives an estimation. I need to measure (and maybe compare) methods that gives an estimation of the winning chance of a player against an other player. b.) 20-30k games by 150 players (so around 200-400 games for each player). – mimrock Jun 19 '14 at 15:45
• What methods do you want to compare? Or do you want to develop a new method? – Joel W. Jun 19 '14 at 16:40
• ELO, Glicko, Glicko-2, etc. I was also thinking about creating new methods, or modifying an existing once - I need something that can measure the accuracy of any method which gives an estimations on the winning chances. – mimrock Jun 19 '14 at 16:54
• It is possible to calculate a correlation coefficient between a continuous and a binary variable. Read up a bit on correlation coefficients. Good luck. – Joel W. Jun 20 '14 at 10:31