Comparing Binomial samples (probability)

Question

I'll try to make this as to the point as possible.

Consider 2 people playing a few games against a computer.
Player 1 gets 6/10 wins
Player 2 gets 56/100 wins
What's the chance that player 1 is better than player 2?

This is a question I've been struggling with for a few days now, and I've come across this question, which is completely equivalent: Binomial Probability Question
However, it leaves me with more questions than I came with.

Disclaimer: It's been years since I've had an introductory course to statistics. I've tried hard to look up answers, but most of it is going over my head. (I knew nothing about Bayesian statistics yesterday, and just a little now.)

Questions:

• In the link I provided, they use Bayesian statistics. Is there a reasonable frequentist way to solve this problem? (I might understand such an approach better)

I tried to find a Probability Density Function for the "true" win percentage of each player. A section in my old textbook, Confidence interval for a proportion, looked promising. However, this relies on the fact that the samples need to be sufficiently large, which is not the case here.

• In the linked answer, the integrals come a bit out of nowhere (at least for me). Could someone explain how they came to the posterior probability distribution in that first expression?

Also, the last step

We then need to integrate that over the probability distribution for John's free throw percentages.

seems to be the key. I think I have a general understanding of what is happening, but if a nice (graphical) explanation of this is available online, I'd also be really interested.

---

EDIT:

I think I get it now ... A more general form of the posterior probability is

$\frac{r^{n+\alpha-1}(1-r)^{N-n+\beta-1}}{\int_0^1 r^{n+\alpha-1}(1-r)^{N-n+\beta-1} dr}$

But since they used an uninformative prior, $\alpha = \beta = 1$, it simplifies to
$\frac{r^{n}(1-r)^{N-n}}{\int_0^1 r^{n}(1-r)^{N-n} dr}$

Now the rest makes perfect sense as well. Thanks for getting me back on track.

Answer 1

To tackle the problem from the Bayesian perspective, you first need to define the priors of the win rates (skill) of the players. Since we are dealing with 0-1 outcomes (loss/win) in this case, it is natural to use Bernoulli distribution as the sampling distribution of the wins and losses. Now, there is a convenient choice for the prior and that is the Beta distribution (why convenient - see the notes about conjugate priors).

Let's define the sampling distribution of the game outcomes for the $i$-th player as

$$P_i(x|\theta_i) = \theta_i^x(1-\theta_i)^{(1 - x)}$$

where $x\in\{0,1\}$ (0 - loss, 1 - win). Let's define the prior distribution of the $i$-th player skill as

$$ B_i(\theta_i;\alpha_i,\beta_i) \propto \theta_i^{\alpha_i-1}(1 - \theta_i)^{\beta-1} $$

Assuming the game plays are independent of each other, you will find out that the posterior probability of $i$-th player is yet another Beta distribution (due to the mentioned conjugacy)

$$ B(\theta_i;\alpha'_i,\beta'_i) = B_i(\theta_i;\sum_{j=1}^{N_i}x_j^{i} + \alpha_i, \left(\sum_{j=1}^{N_i}1 - x_j^{i}\right) + \beta_i) = \prod_{j=1}^{N_i} P_i(x^j_i|\theta_i)B_i(\theta_i;\alpha_i,\beta_i) $$

where $x^i_j$ is the outcome of the $j$-th game of the $i$-th player in total of $N_i$ games.

Now we are almost there. Lets say you knew for certain that the 1st player skill is $\phi$ and you hold a belief about the skill of the 2nd player as $B(\theta;\alpha,\beta)$ then the probability of the 2nd player being superior to the 1st is given by $$ \int_{\phi}^1B(\theta;\alpha,\beta)d\theta $$ but since in your case you do not know the skill of the 1st player, but you have your own belief about it, you apply the Bayesian "integrate over what you do not know" approach which gives you the final expression $$ \int_0^1\int_{\theta_1}^1B(\theta_1;\alpha'_1,\beta'_1)B(\theta_2;\alpha'_2,\beta'_2)d\theta_1{}d\theta_2 $$

The choice of the prior $\alpha$a and $\beta$s is actually up to you and you should actually notice that they have interpretable meaning. Use $\alpha=\beta=1$ for instance to get a uniform prior.

If you know Python by any chance, you may actually code a pretty simple Monte Carlo simulation to get an approximate answer two this integral (where I used uniform priors):

import numpy as np
num_samples = 1000000
player_1 = np.random.beta(7, 5, size=num_samples)
player_2 = np.random.beta(57, 45, size=num_samples)
print np.mean(player_1 > player_2)

>> 0.574937

I am pretty much sure even if you do not know Python the code is pretty much self-explanatory. Hope this helps clarify some things up :).