Test whether player X is better than Player Y

Question

Player A has beaten player B 3 times in a game with some amount of luck, random card draw. What is the probability that player A is in fact the better player how could we extend that to X tires?

Answer 1

If you mean A won all three of three games, in which it is reasonable to say that A and B had equal opportunity to win, then an exact binomial test should tell you whether the null hypothesis of equal skill should be rejected. (Notes: An approximate normal test would not be accurate with such a small number of trials. Outside of a Bayesian approach, I don't think you can get a probability that A is better than B; but you can to a test.)

binom.test(3, 3)

        Exact binomial test

data:  3 and 3
number of successes = 3, number of trials = 3, p-value = 0.25
alternative hypothesis: 
 true probability of success is not equal to 0.5
95 percent confidence interval:
 0.2924018 1.0000000
sample estimates:
 probability of success 
                      1 

Perhaps more simply: with three games, there are eight possible outcomes. For a 2-sided test either AAA or BBB would be the most extreme outcomes, that gives P-value $2/8 = 0.25.$ If you somehow suspected, in advance of seeing any results, that A is the more-skilled player, then you might do a one-sided test, rejecting the null hypothesis only upon an AAA result. Then the P-value would be $1/8$--- still larger than 5%. (There is not enough information in three plays of the game to get a convincing test result.)

You might wonder, how many plays of the game it would take for a two-sided test to be able to reject at the 5% level, if A won every time. That is, you want the P-value of the two-sided test to be less than or equal to $0.05 = 5\%.$ You should be able to get the answer by simple combinatorial analysis as above. But if you want to see the test results, their P-values are shown below. Six games would suffice.

binom.test(5, 5)$p.val
[1] 0.0625
binom.test(6, 6)$p.val
[1] 0.03125