Statistically significant win rate I'm playing around with AIs for Onitama, a 2 player pefect knowledge game.
I want to compare the strengths of different AIs by measuring their wins and loses and then testing for statistical significance.



*

*The results are from the perspective of the row

*"draw" really means "50 turns with no winner")

*"min_max_X" means "min_max searching to depth X"


My first attempt is to use a binomial test.
Ignoring draws, my null hypothesis is that equal strength AIs win 50% of the time.
I calculated P-values using scipy
p_value = stats.binom_test(x=result["win"], n=result["win"] + result["lose"], p=0.5, alternative='two-sided')


These results seem reasonable to me, assuming I'm interpreting right:


*

*The closer the P-value is to 0, the more likely null-hypothesis is to be wrong

*"random" has low P-values against min-max, as expected

*For AIs playing themselves, we get high P-values, as expected


Is my approach valid for this situation, and is there a better/more common way?
 A: I tried several of these computations in R using binom.test, obtaining results that match yours.
In particular, for 'Random vs Random', I got:
binom.test(21, 21+17, alt="two")

        Exact binomial test

data:  21 and 21 + 17
number of successes = 21, number of trials = 38, p-value = 0.6271
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.3829908 0.7137585
sample estimates:
probability of success 
             0.5526316 

Also, for 'min_max3 va min_max2':
binom.test(21, 21+10, alt="two")

        Exact binomial test

data:  21 and 21 + 10
number of successes = 21, number of trials = 31, p-value = 0.07076
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.4862702 0.8331764
sample estimates:
probability of success 
             0.6774194 


Sometimes, this is called a 'sign test' where Wins are called +, Losses -, and Draws are ignored. For the first example above ('Random vs Random'),
the P-value for a two-sided test can be computed, by symmetry, as $2P(X \le 17)= 0.6271,$ where
$X \sim \mathsf{Binom}(n=38,\, p=1/2).$
2*pbinom(17, 21+17, 1/2)
[1] 0.6271026


