# How do you verify a rating system?

Let's say I have a database of a lot of (thousands) chess games. In chess, every player have a rating, called Elo that gives an estimation how strong is this player. If we know the the Elo ranking of both players, we can give an estimation of the probabilities of the possible outcomes of the match. In theory, a player whose rating is 100 points greater than their opponent's is expected to score 64%.

This is theory. The player's true skill can be different. E.g. if he/she is really tired, or out of practice, his/her true skill will be lower than his/her Elo. So for every game I have an estimation (if we don't count the draw, it's pratcically a simple binomial experiment) of the probability of the different outcomes, and the outcome itself. Is there a statistic method, with which I can verify how accurate is this estimation?

I need something that can tell how accurate is the method of estimation not just disprove a null hypotesis that it doesn't have anything to do with player's true skills.

• Perhaps you can clarify a bit: (a) What is the relationship between players' ratings and expected winning of a match? (b) How much data do you have? Jun 19, 2014 at 13:32
• (a) The higher the rating the more chance of winning. However, it's not the rating that counts. It's the estimated probability of winning. ELO rating is just an example which gives an estimation. I need to measure (and maybe compare) methods that gives an estimation of the winning chance of a player against an other player. b.) 20-30k games by 150 players (so around 200-400 games for each player). Jun 19, 2014 at 15:45
• What methods do you want to compare? Or do you want to develop a new method? Jun 19, 2014 at 16:40
• ELO, Glicko, Glicko-2, etc. I was also thinking about creating new methods, or modifying an existing once - I need something that can measure the accuracy of any method which gives an estimations on the winning chances. Jun 19, 2014 at 16:54
• It is possible to calculate a correlation coefficient between a continuous and a binary variable. Read up a bit on correlation coefficients. Good luck. Jun 20, 2014 at 10:31

## 1 Answer

Probably most famous example for testing how accurate is the method of estimation in chess rating system was Chess ratings - Elo versus the Rest of the World competition on Kaggle, which structure was the following:

Competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games.

Winner was Elo++.