How many Backgammon dice rolls to make doubles percentage statistically significant?

Question

I run an online Backgammon game and the most common complaint that I get is that the computer rolls too many doubles (a complaint for every online Backgammon game apparently). I started tracking the doubles so people can see they get as many as the computer, but people keep telling me that the computer gets more than them, even after only a few rolls.

There are two dice, there are 36 outcomes, 6 of which are doubles, so there should be a 6/36 = 16.6% chance of rolling a double. So, if someone says the computer rolls 20% doubles and they only roll 12% doubles, out of X rolls, what does X need to be for me to conclude that there's something wrong with the dice rolling? And, after a lot of rolls, say 50000, what would be a normal variance from the 16.6% ? Would 15% still be normal?

Answer 1

This is very suitable for Hypothesis Testing, and there already exists a framework for Testing Difference in Proportions. First, we formulate our hypotheses:

$$H = \begin{cases} H_0: P_c=P_h \\ H_A: P_c\neq P_h \end{cases} $$

($P_c, P_h$ are computer and human proportions respectively.) This is two-tailed test for the equality of the two proportions. There is also a one-tailed test for testing if one is greater than the other or else. I think, for a fair game, you should test for equivalence. But, if you keep getting $P_c > P_h$, you can also consider having a one-tailed test.

First of all, you compute a pooled sample proportion, and standard error, using your reported computer/human proportions/# of throws:

$$p_{pool}=\frac{p_cn_c+p_hn_h}{n_c+n_h}$$ $$S=\sqrt{p_{pool}(1-p_{pool}) \left( \frac{1}{n_c}+\frac{1}{n_h} \right)}$$

Then, calculate your test statistic: $z_{cal}=\frac{p_c-p_h}{S}$

Finally, you check if this value is outside your significance level thresholds. A common significance level would be $\alpha=0.05$, which correspond to range $[-1.96,1.96]$ according to standard normal table, i.e. if your calculated value is outside this range, then it means your have high chance of having biased dice throws.

For single value evaluation, i.e. questions like "is it normal to have 0.15 ?" etc; we employ a very similar technique:

$$\sigma=\sqrt{\frac{p_{true}(1-p_{true})}{n}}, z_{cal}=\frac{p_{true}-p_{observed}}{\sigma}$$, And similarly compare this calculated $z$ value with $\alpha$ significance level thresholds. For example, for $n = 50000, p_{observed}=0.15, p_{true}=0.166$ would have $z_{cal}\approx 9.62$, which is very unusual.