test if 5% difference is significant

Question

I have a dataset that contains 20000 chess matches. According to these matches, white won 50% of the games while black won 45%. I want to test if this 5% difference is significant. How can I do that? I couldn't be sure which testing method to use.

Answer 1

The white won 10000 matches out of 20000. The black won 9000 matches out of 20000.

You can carry out a two-proportion z-test. For instance in R:

result <- prop.test(x = c(10000, 9000), n = c(20000, 20000))

Output:

2-sample test for equality of proportions with continuity correction

data:  c(10000, 9000) out of c(20000, 20000)
X-squared = 100.05, df = 1, p-value < 2.2e-16
alternative hypothesis: two.sided
95 percent confidence interval:
 0.04017471 0.05982529
sample estimates:
prop 1 prop 2 
  0.50   0.45

The p-value is < 2.2e-16 so the difference is statistically significant.

(You can replace 20000 by 19000 if you only want to consider the matches won by either the white or black).

Answer 2

Suppose you have data for 10,000 matches. I use R to simulate results according to your 50% (1=white), 45% (2=black), 5% (3=draw) hypothesis.

set.seed(131)
x = sample(1:3, 10000, rep=T, p=(.50,.45,.05))
table(x)
x
   1    2    3 
5029 4499  472 

In prop.test you can specify the hypothetical probabilities, as shown below.

prop.test(c(5029,4499,472), c(10000, 10000, 10000),
          c(.5,.45,.05))

    3-sample test for given proportions 
    without continuity correction

data:  c(5029, 4499, 472) out of c(10000, 10000, 10000), 
  null probabilities c(0.5, 0.45, 0.05)
X-squared = 1.9873, df = 3, p-value = 0.575
alternative hypothesis: two.sided
null values:
prop 1 prop 2 prop 3 
  0.50   0.45   0.05 
sample estimates:
prop 1 prop 2 prop 3 
0.5029 0.4499 0.0472 

So the observed proportions were $0.5029, 0.4499, 0.0472,$ which are sufficiently close to the hypothetical proportions $.50, .45, .05$ to be considered consistent with the null hypothesis.

Over the long run, deviations (from hypothetical proportions) as large as seen in our simulation occur with probability about $.58,$ and so are not considered unusual.

Testing at the 5% level of significance (rejecting when the P-value is smaller the 0.05), we can expect this simulation experiment to give values inconsistent with the null hypothesis only 5% of the time.