6
$\begingroup$

A definitive proposal can be found in:

Elo, Arpad E. (August, 1967). The Proposed USCF Rating System, Its Development, Theory, and Applications. Chess Life XXII (8): 242-247.

How exactly are

  • THE PERCENTAGE EXPECTANCY CURVE (Fig. 1),
  • TABLE I - RATING DIFFERENCES, and
  • TABLE II- WINNING EXPECTANCIES

(page 243-244) calculated, including rounding details. I tried the normal distribution function with mean = 0 and sigma = 2000 / 7. But I could not get the rounding correct. For example the percentage expectancy of rating difference DP = 344 is 89% instead of 88% as in table II. Other irregularities: 54, 343, 358, 392, 620.

At page 247 we find a short reference list, including:

  • A. E. Elo; Analytical Reports, 1961, 1963 & 1965; (Privately Printed).

Are these reports lost forever, or are copies preserved and available in the public domain?

Update

My question is not about the validity of the model, but about the calculation within the model. How exactly can I reproduce Table II? Why is the rating expectancy of DP = 344 equal to P = .88 instead of .89 as I would expect.

In The Rating of Chess players, Elo states in chapter 8.94 Development of the Percentage Expectancy Table:

For example, suppose D = 160. Then z = 160/284.84. The table gives .7143 and .2857. as the areas of the two portions under the curve. These probabilities are rounded to two figures in table 2.11.

Apparently Elo refers to a z-table, but which z-table was in use at that time (±1960)?

  • @whuber I tried.
        σ=2000/7
  D   H |z-score |  Tabel (t)         |  t/2 + 50%      |  N(z)           | P
 54  58%| 0,189  |0.14990 71832 09998 |0,574953591604999|0,574953591604999| 57%

        | 1,200  |0.76986 06595 56583 |0,884930329778292|0,884930329778292|
 343 88%| 1,2005 |                    |  interpolatie   |0,885027364557095| 89%
        | 1,201  |0.77024 87986 71795 |0,885124399335897|0,885124399335898|

 344 88%| 1,204  |0.77141 04214 54696 |0,885705210727348|0,885705210727348| 89%
 358 90%| 1,253  |0.78979 42933 03286 |0,894897146651643|0,894897146651643| 89%
 392 92%| 1,372  |0.82993 65621 86873 |0,914968281093437|0,914968281093436| 91%
 620 99%| 2,170  |0.96999 31540 52536 |0,984996577026268|0,984996577026268| 98%

Source:

(Issued June 5, 1953). Tables of Normal Probability Functions U. S. Applied Mathematics (23) (Department of Commerce National Bureau of Standards).

  • @cdalitz, 2000/7 is folklore, used by the Dutch Chess federation. I have no reference from literature.

Constructing TABLE-1 from TABLE-2

Table I and II are effectively mirror-images. See also the FIDE Handbook B. Permanent Commissions / 02. FIDE Rating Regulations.

R-code

D <-
  c(  3,  10,  17,  25,  32,  39,  46,  53,  61,  68,  76,  83,  91,  98, 106, 113, 121,
    129, 137, 145, 153, 162, 170, 179, 188, 197, 206, 215, 225, 235, 245, 256, 267, 278,
    290, 302, 315, 328, 344, 357, 374, 391, 411, 432, 456, 484, 517, 559, 619, 735, 736
   )

## TABLE I - RATING DIFFERENCES, between 51% and 99%.
## Column DP of TABLE-1 constitutes the midpoints of TABLE II, indexed by P.
## This is a reasonable approximation with the exception of P=99%.
table_1 <-
  head(cbind(P =seq(from=0.51, to=1.0, by=0.01),
             DP=trunc((head(D, -1) + tail(D, -1) + 1)/2) ),
       -1)
table_1

Constructing TABLE II - WINNING EXPECTANCIES with sigma = 200 * (10/7)

Note (10/7) is a rough approximation of sqrt(2). Conversion of rating difference to z-score gives at most 4 significant digits, which is useful when using a table indexed by z-score.

## Create rating differences, winning expectancy between 1 and 736, from 50% upto 100%.
n     <- 736
Pfull <- rep(-1, n)
pct   <- 0.50
k     <- 1
for(i in seq_along(D) ){
  while(k <= D[i]){
    Pfull[k] <- pct
    k        <- k + 1
  }
  pct <- pct + 0.01
}

## calculate / add normal probabilities (pnorm).
# P2, P15 probabilities rounded by 2, 15 digits.
df <- data.frame(D=seq(n), P=Pfull)

Deviations in Elo's expectations table

sd <- 200 * (10/7)  # standard deviation, using (10/7) instead of sqrt(2).
within(df,
{ P15 =  round(pnorm(D / sd , 0, 1,1), 15)
  P2  = round(P15, 2)
  attP2  = round(P - P2,  10)
}
) -> df2
subset(df2, attP2 != 0)
      D    P attP2   P2       P15
54   54 0.58  0.01 0.57 0.5749536
343 343 0.88 -0.01 0.89 0.8850274
344 344 0.88 -0.01 0.89 0.8857052
358 358 0.90  0.01 0.89 0.8948971
392 392 0.92  0.01 0.91 0.9149683
620 620 0.99  0.01 0.98 0.9849966

Constructing TABLE II by interpolation, third try

Consider rating values in steps of 20. The corresponding Z-score step is 20 * 7 / 2000 = 0.07. Finding the percentage scores corresponding to the rating values D = 0, 20, 40, etc. does not require any calculations. They can be extracted directly from the Z-table.

To calculate the intermediate values, linear interpolation is the most obvious. For example calculate the P value of 54 as follows:

sd  = 2000 / 7, the standard deviation chosen to ensure discrete steps in the Z table.
D   = 40 + 14, where 14 is the index in rating range 40-60
Z_b = 40 * 7 / 2000 = 0.14,  P_b = 0.5557 (From the Z table)
Z_e = Z_b + 0.07    = 0.21   P_e = 0.5832 (From the Z table)

The step size of P becomes: (0.5832 - 0.5557) / 20 = 0.001375

This gives:

P4  =  0.5557 + 14 * 0.001375 = 0.57495
    = 0.57495 + 5E-5 + 1E-6, rounded up to two decimals gives
P42 = 58%.

R-code.

dD <- 20                                          # step in D = 0, 20, 40, etc.
sd <- 2000 / 7                                    # standard deviation.
dZ <- dD / sd                                     # Short table, Z in two decimals.
within(df,
{ Z_sc    <- D / sd                               # 4 decimals.
  P15     <- round(pnorm(Z_sc), 15)               # value from extended table (1953).       
  Z_b     <- trunc(D / dD) * dD * 7 / 2000        # index in short table, begin.
  Z_e     <- Z_b + dZ                             # index in short table, end.
  P_b     <- round(pnorm(Z_b ), 4)                # P before.
  P_e     <- round(pnorm(Z_b + dZ), 4)            # P after.

  ##  Corrections to fit Elo tables.
  ## P_b[340:359] <- .8830 - .0010                  # Feller = .8836 
  ## P_b[380:399] <- .9082 + .0001                  # Feller = .9083 

  ix      <- D %% dD                              # index 0 - 20.
  Pstep   <- (P_e - P_b) / dD                     # interpolation step.

  ## Pstep[340:359] <- Pstep[320:339]               # Previous step size
  ## Pstep[380:399] <- Pstep[380:399] + .00001      # .00545 --> .00555

  P4e     <- P_b + ix * Pstep                     # intermediate percentage.
  P42e    <- round(P4e + 5E-5 + 1E-6, 2)          # rounded up to next percentage.
  attP42e <- round(P42e - P, 15)                  # deviation .   
}
) -> inter_lin
#subset(inter_lin, D >= 340 & D <= 360)
subset(inter_lin, attP42e != 0)

Between D = 340 and D = 400 we find 4 deviations:

 D    P attP42e P42e     P4e   Pstep ix    P_e    P_b  Z_e  Z_b       P15   Z_sc
343 0.88    0.01 0.89 0.88498 0.00066  3 0.8962 0.8830 1.26 1.19 0.8850274 1.2005
344 0.88    0.01 0.89 0.88564 0.00066  4 0.8962 0.8830 1.26 1.19 0.8857052 1.2040
358 0.90   -0.01 0.89 0.89488 0.00066 18 0.8962 0.8830 1.26 1.19 0.8948971 1.2530
392 0.92   -0.01 0.91 0.91480 0.00055 12 0.9192 0.9082 1.40 1.33 0.9149683 1.3720

How did it happen? It is easy to see that the table is not entirely consistent. Look at the number of rating differences assigned to a percentage:

  P  L  -  H   Range
.86 303 - 315   13 
.87 316 - 328   13 
.88 329 - 344   16 <<<
.89 345 - 357   13
.90 358 - 374   20

Since the expectation function is monotonically increasing, the difference between H and L should not decrease. The 16 is clearly an outlier.

The error in D = 392 can be explained by using .00555 instead of .00545 as the step size for the linear interpolation. Errors in D=343, 344, 358 can be explained by taking a wrong starting percentage (related to Z = 1.19), and the step size of the previous group.

The above is obvious an educated guess. I am afraid I have to visit the Royal Library in the Hague to find out more.

I am considering moving my question to the Chess stackexchange site. But I don't know what the best approach is yet.

$\endgroup$
15
  • $\begingroup$ Where did you read that $\sigma=2000/7$? Towards the end of p. 243 (rightmost column), Elo writes of $\sigma=200$. $\endgroup$
    – cdalitz
    Jun 19, 2023 at 13:49
  • $\begingroup$ "which z-table was in use at that time (±1960)?" The article is from 1967, so the classic reference book for tables would have been Abramowitz, Stegun: "Handbook of Matehmatical Functions" from 1964. $\endgroup$
    – cdalitz
    Jun 19, 2023 at 13:53
  • $\begingroup$ I have read about $200 \sqrt{2}\approx 2000/7$ before here: stats.stackexchange.com/questions/477721/… (edit: ah, that was you) $\endgroup$ Jun 19, 2023 at 13:59
  • 1
    $\begingroup$ A few sources that deal with the 2000/7 deviation mention a 1966 book by Elo named "the theory of rating systems". An online source could be the appendix in An Application of the Elo Rating System to Professional Baseball which would be available to students from Michigan. $\endgroup$ Jun 25, 2023 at 23:15
  • 2
    $\begingroup$ I would like to suggest it's time to stop editing this post: they have become trivial and are likely to annoying users by now, because each time it bumps the thread to the top of the list of active ones. $\endgroup$
    – whuber
    Jul 3, 2023 at 14:11

3 Answers 3

5
$\begingroup$

copy of tables

The values in table I seem to be an average of the values in table II (rather than a direct computation with the quantile function).

The values in table II correspond very well with a normal distribution that has standard deviation $2000/7$. The upper boundaries will be equal to the percentile scores of the normal distribution rounded down and the lower boundaries will be equal to the percentile scores of the normal distribution rounded up. Below you see a comparison of three methods.

example

The logistic distribution or a normal distribution with standard deviation $200 \sqrt{2}$ have large differences.

For the normal distribution with standard deviation $2000/7$ there are some small discrepancies, but only of the order of $\pm 1$ and that can be due to round-off errors and use of tables with lower precision* or possibly a typo.

p = seq(0.505,0.995,0.01)
elo1 = qnorm(p,0,2000/7)
elo2 = qnorm(p,0,200*sqrt(2))
elo3 = -log10(1/p-1)*400

x = c(3,10,17,25,32,39,46,53,61,68,76,83,91,98,106,113,121,129,137,145,153,162,170,179,188,197,206,215,225,235,245,256,267,278,290,302,315,328,344,357,374,391,411,432,456,484,517,559,619,735)

plot(x,x-x, type = "l", xlab = "Elo's upper boundary", log = "", ylim = c(-2,15), ylab = "difference with table")
points(x,elo1-x, pch = 20)
points(x,elo2-x, pch = 2)
points(x,elo3-x, pch = 4)

legend(1,15, c("normal 2000/7", "normal 200*sqrt(2)", "logistic distribution"), cex = 1, pch = c(20,2,4))

lines(x,x*0+1)

In 'The rating of chess players, past and present' Elo writes an example

For example. let D = 160. Then z = 160/282.84 = .566. The table gives .7143 and .2857 as the areas of the two portions under the curve. These probabilities are rounded to two figures in table 2.11

This suggests that he used sufficiently accurate tables with at least 4 figures. An example could be Table X (copied from Pearson) in Garrett's 'statistics in psychology and education' which occurs as a reference.

$\endgroup$
1
  • $\begingroup$ Table I is derived from table II. So it is about table II. Most values are only one rating point off, with the exception of 343, 344. Elo is rather vague over the origin of his table. Is it possible to reconstruct his calculation from old sources? I tried simulating tables with less precision, but nothing worked. $\endgroup$
    – clp
    Jun 19, 2023 at 15:18
11
$\begingroup$

There is a book by Elo entirely devoted to his formulas:

Arpad. E. Elo: "The Rating of Chessplayers." Ishi Press, 1978

Although the book presents some formulas, the derivation is not argued in details, and some formulas only hold under strong (rarely holding) assumptions, and it is not even discussed what these assumptions are or whether the estimators are biased or consistent. Just as one example, consider the "performance rating formula" (1) in your cited article. It can be derived by the "method of moments", i.e. by equating an expected value with its observed value as follows:

  1. If a player of unknown rating $r$ plays against $n$ players with ratings $q_1,\ldots,q_n$, then the expected number of wins $W$ is $$E(W) = \sum_{j=1}^n \Phi(r-q_j)$$ where $\Phi$ is the normal distribution function with mean zero and variance $\sigma^2$.
  2. Setting the observed number of wins $\hat{W}$ equal to the expected number of wins in the above formula yields an equation for $r$, that cannot be solved in closed form.
  3. For solving this equation, Elo makes the assumption (without even mentioning it) that $\sum_j \Phi(x_i) = \Phi\left(\sum_i x_i\right)$ and obtains $$\Phi^{-1}\left(\frac{\hat{W}}{n}\right) + \frac{1}{n} \sum_j q_j = r$$ which is the "performance rating formula" (1).
  4. Obviously, Elo's assumption does not hold unless $\Phi$ is a linear function, and at some later point (p. 133, equation(18) in the 1978 book above) he says that $\Phi(x)$ may be taken as $x/4\sigma$ "for the most used portion of the curve". Unfortunately, he did not explain how he arrived at this approximation. A Taylor expansion of $\Phi$ around $x=0$ would yield $\Phi(x)\approx x/\sqrt{2\pi}\sigma $, and $\sqrt{2\pi}$ is considerably different from 4, so it is not clear how Elo derived his approximation. Fortunately, this (presumable) error does not matter much, because he later subsumes the factor $4\sigma$ together with other factors in a value $K$, for which he suggests two different choices, depending on the number $n$ of played games. In any case, the linear approximation only holds when the ratings of all players are quite close to each other.

For a more lucid presentation of the mathematical model and an accurate derivation of estimation formulas (which are different from Elo's formulas), see

Batchelder, William H., and Neil J. Bershad. "The statistical analysis of a Thurstonian model for rating chess players." Journal of Mathematical Psychology 19.1 (1979): 39-60.

The discussed model differs from Elo's model, though, in the additional assumption that a chess game can result in ties (Elo only uses a model with two possible outcomes: win or loss, and draws are later counted in the formulas as half wins, which means that you cannot compute the probability of a draw with Elo's model). Elo's model can be obtained, however, as a special case of Batchelder's and Bershad's model simply by assuming that a tie is impossible and thereby setting the "tie width" $t$ to zero.

To address your question what the three terms mean in the context of the model, we must frst define the model: $$P(\mbox{i wins against j})=\Phi(r_i - r_j)$$ where $\Phi$ is the distribution function of the normal (or alternatively: logistic) distribution with zero mean and standard deviation $\sigma$. In the cited article Elo alleges that he uses $\sigma=200$, but this does not match the tables listed in his article. Table I should be $\Phi^{-1}$ and Table II $\Phi$. I obtain approximately similar numbers with the choice $\sigma=286$, e.g.

> # reproduce values from Table I
> qnorm(0.60, sd=286)
[1] 72.45727
> qnorm(0.19, sd=286)
[1] -251.0783

or

> # reproduce values from Table II
> pnorm(210, sd=286)
[1] 0.7686066
> pnorm(95, sd=286)
[1] 0.6301187

which approximately matches the numbers in the tables. The alleged choice of $\sigma=200$ is also inconsistent with the use of a logistic distribution:

> plogis(220, s=200*sqrt(3)/pi)
[1] 0.88029

which is very far from the number given by Elo.

$\endgroup$
3
  • $\begingroup$ My question is not about the validity of the model, but about calculations within the model, regardless of interpretation. How to reproduce? $\endgroup$
    – clp
    Jun 19, 2023 at 13:09
  • $\begingroup$ @clp You are right that the tables in Elo's article do not match his description what the numbers allegedly are. It seems that he used a different choice for $\sigma$ for computing the tables. $\endgroup$
    – cdalitz
    Jun 19, 2023 at 13:41
  • $\begingroup$ Elo uses σ = 200 as the USCF class interval for a single player, and 200sqrt(2) for a single game. That is: 282.84. In reality, however, he used 2000 / 7 (almost 286). Why? I think because 1 / σ gives only at most 4 significant decimal places. This is useful when looking up values in tables. I assume when the table was created (before 1960) he did not yet have a pocket calculator. Are the few differences inaccuracies or is there a reasonable explanation? To be found in the missing documents? $\endgroup$
    – clp
    Jun 19, 2023 at 14:05
4
$\begingroup$

Elo was specific: "This basic relationship ... turns out to be the familiar cumulative proportion curve of probability theory, the 'standard sigmoid'." That's an old-fashioned way of saying that differences on the rating scale are linearly related to Z-scores: standard Normal deviates.

Here is his plot of that relationship for differences of scores, which therefore are nominally $200\sqrt{2}\approx 283$ per standard deviation unit:

enter image description here

This evidently is hand drawn. Here is my plot of the same, but using a scale of $202.3\sqrt{2}$ because that matches the tables better:

enter image description here

The dots plot the values from Table 1. The entire point is that on this Normal probability plotting scale, the relationship between ratings and probability is linear.

Plotting the difference between the rating difference "$d(P)$" and the rating as computed from the Normal distribution from the probability "$P$" (the residuals in the foregoing plot) suggests Elo used an approximation that works poorly in the tails (beyond the central 95% of the distribution, more or less). (More detailed study suggests this is a severely truncated rational approximation.) The worst discrepancy is at the end, where the score difference of 677 really should be 666.

Those who want details might be interested in the code that generated this figure.

#
# Table 1.
#
d <- c(677, 589, 538, 501, 470, 444, 422, 401, 383, 366, 351, 336, 322, 309, 296, 284,
       273, 262, 251, 240, 230, 220, 211, 202, 193, 184, 175, 166, 158, 149, 141, 133, 125,
       117, 110, 102, 95, 87, 80, 72, 65, 57, 50, 43, 36, 29, 21, 14, 7, 0)
p <- seq(1.0, 0.50, length = length(d) + 1)[-1]
X <- data.frame(P = p, Delta = d)
#
# The transformations relating "P" to "d(P)" in the table.
#
F. <- function(x, lambda = 202.3 * sqrt(2)) pnorm(x / lambda)
Q. <- function(p, lambda = 202.3 * sqrt(2)) lambda * qnorm(p)
X$Q <- round(Q.(X$P), 0)
X$Check <- with(X, Delta - Q)
with(X, plot(Delta, round(Q.(P) / Delta - 1, 3)))
with(X, plot(Delta, Q.(P) - Delta))
X$F <- round(F.(X$Delta), 3)
with(X, plot(P, round(F.(Delta) - P, 3)))
#
# Replot Figure 1.
#
y <- c(.01, .1, .5, 1, 2, 5, 10, 20, 30, 40)
y <- c(y, 50, 100 - rev(y)) / 100                        # The y grid
x <- seq(0, 900, by = 100)                               # The x grid

library(ggplot2)
X <- subset(X, P < 1)
scale.USCF <- 202.3
rho <- 4/3 * diff(range(x)) / diff(range(qnorm(y)))      # 4/3 is aspect ratio
alpha <- atan(rho / scale.USCF * sqrt(2) / 2) * 180 / pi # Angles of the lines

ggplot(data.frame(x = x)) +
  geom_vline(aes(xintercept = x), col = gray(0.7)) +
  geom_hline(aes(yintercept = y), data = data.frame(y = y), col = gray(0.7)) +
  geom_abline(slope = c(-1, 1) * 1 / (scale.USCF * sqrt(2))) +
  geom_point(aes(Delta, P), data = X, col = hsv(0.05, 1, .8)) + 
  geom_point(aes(Delta, 1 - P), data = X, col = hsv(.6, 1, .8)) + 
  coord_fixed(ylim = range(y), xlim = c(0, 920), expand = FALSE, ratio = rho) +
  scale_y_continuous(trans = scales:::probability_trans("norm"), breaks = y) +
  scale_x_continuous(breaks = x) + 
  annotate("text", x = rep(400, 2), y = pnorm(c(1.58, 0.36 - 1.58)),
           label = paste(c("Higher", "Lower"), "Rated Player"),
           angle = alpha * c(1, -1), size = 3.5) +
  xlab("Difference in Rating (USCF Scale)") +
  ylab("Probability of Winning a SingleGame") +
  ggtitle("The Percentage Expectancy Curve", "After ELO (1967) Figure 1") + 
  theme(panel.grid = element_blank(),
        panel.background = element_blank())
$\endgroup$
5
  • $\begingroup$ Why would Elo use an approximation or graphical method when tables of that time give highly accurate results? In "The rating of chess players, past and present" he writes an example "For example. let D = 160. Then z = 160/282.84 = .566. The table gives .7143 and .2857 as the areas of the two portions under the curve. These probabilities are rounded to two figures in table 2.11" which suggests that he used tables that gave at least 4 figures. An example could be Table X (copied from Pearson) in Garrett's "statistics in psychology and education" $\endgroup$ Jun 25, 2023 at 22:23
  • $\begingroup$ A very good question (not an answer). The point is that he apparently did not, at least not consistently. Try D=25. Then DP=54%. However, the table (p. 176) shows 53%. I am using the table from Feller's An Introduction to Probability Theory and its Applications, Z in two figures, P in four figures. $\endgroup$
    – clp
    Jun 26, 2023 at 15:51
  • $\begingroup$ In those days many people did calculations by hand -- literally, pencil and paper. It is not at all unusual to see inconsistencies, imprecision, the unstated use of not so good approximations, and so on. Because the approach is clear and has been superseded by later writings, there seems little point to trying to reproduce either of these tables exactly. $\endgroup$
    – whuber
    Jun 26, 2023 at 15:54
  • $\begingroup$ These tables have never been corrected and are still in use. $\endgroup$
    – clp
    Jun 26, 2023 at 16:12
  • $\begingroup$ Then you might try to correct them. Speculating about why they might be imprecise doesn't really help with that. I say "try" because sometimes, even when a procedure is not completely correct, it can become a standard or part of a law or so deeply embedded within a complex system that it shouldn't be corrected. But that's a matter for chess authorities to consider. $\endgroup$
    – whuber
    Jun 26, 2023 at 16:28

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .