Statistical test for player strategies results (win/loss/draw)

Question

What is the suitable statistical test for checking statistics significance and confident intervals for this type of data?

I have several agents (strategies) to play a game, and the result of each game end is recorded (see the attached image as a small sample of my data. I have thousands of such results).

At the end I'm counting the times each agent (Black=First Player) won, loss or did a draw against all the others.

For example strategy X (as Black player) has:

• 50% wins(4/8)
• 25% Losses (2/8)
• 25% Draws (2/8)

My null hypothesis will usually be that the agents have equal strength and will therefore win an equal number of games. I want to use a statistical test to test if either player is better.

Answer 1

For each pair of strategies (x,y) you can easily test your hypothesis using CLT for proportions (see for example here). Your null hypothesis would be "no strategy is better" or p = 0.5, and your alternative hypothesis may be p is smaller / larger / different than 0.5.

Since you have n strategies, we need to better define what a 'better strategy' is. If you carefully select a sample of kn(n-1) games, where there are k games for each pair of strategies, then define n subgroups, where subgroup s contains all the k*(n-1) games of strategy s, a null hypothesis "strategy s is not better that average" would again translate to p(s) = 0.5 for subgroup s. If your alternative hypothesis is "strategy s is worse / better / different than average" - you may repeat the same test as before.

As for draws, I would count them as half-win half-loss - to keep p(s) = 0.5.

If the null hypothesis is "strategy s is not better than all others" and your alternative hypothesis is "strategy s in better than all others", you may use the method suggested by Dan Nettleton in "Testing for the Supremacy of a Multinomial Cell Probability" (available here) or the method suggested by Alessio Farcomeni in "Testing Supremacy or Inferiority of Multinomial Cell Probabilities with Application to Biting Preferences of Loggerhead Marine Turtles" (available here).