How many times must I roll a die to confidently assess its fairness?

Question

(Apologies in advance for use of lay language rather than statistical language.)

If I want to measure the odds of rolling each side of a specific physical six-sided die to within about +/- 2% with a reasonable confidence of certainty, how many sample die rolls would be needed?

i.e. How many times would I need to roll a die, counting each result, to be 98% sure that the chances it rolls each side are within 14.6% - 18.7%? (Or some similar criteria where one would be about 98% sure the die is fair to within 2%.)

(This is a real-world concern for simulation games using dice and wanting to be sure certain dice designs are acceptably close to 1/6 chance of rolling each number. There are claims that many common dice designs have been measured rolling 29% 1's by rolling several such dice 1000 times each.)

Answer 1

TL;DR: if $p$ = 1/6 and you want to know how large $n$ needs to be 98% sure the dice is fair (to within 2%), $n$ needs to be at least $n$ ≥ 766.

---

Let $n$ be the number of rolls and $X$ the number of rolls that land on some specified side. Then $X$ follows a Binomial(n,p) distribution where $p$ is the probability of getting that specified side.

By the central limit theorem, we know that

$$\sqrt{n} (X/n - p) \to N(0,p(1-p))$$

Since $X/n$ is the sample mean of $n$ Bernoulli$(p)$ random variables. Hence for large $n$, confidence intervals for $p$ can be constructed as

$$\frac{X}{n} \pm Z \sqrt{\frac{p(1-p)}{n}}$$

Since $p$ is unknown, we can replace it with the sample average $\hat{p} = X/n$, and by various convergence theorems, we know the resulting confidence interval will be asymptotically valid. So we get confidence intervals of the form

$$\hat{p} \pm Z \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

with $\hat{p} = X/n$. I'm going to assume you know what $Z$-scores are. For example, if you want a 95% confidence interval, you take $Z=1.96$. So for a given confidence level $\alpha$ we have

$$\hat{p} \pm Z_\alpha \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

Now let's say you want this confidence interval to be of length less than $C_\alpha$, and want to know how big a sample we need to make this case. Well this is equivelant to asking what $n_\alpha$ satisfies

$$Z_\alpha \sqrt{\frac{\hat{p}(1-\hat{p})}{n_\alpha}} \leq \frac{C_\alpha}{2}$$

Which is then solved to obtain

$$n_\alpha \geq \left(\frac{2 Z_\alpha}{C_\alpha}\right)^2 \hat{p}(1-\hat{p})$$

So plug in your values for $Z_\alpha$, $C_\alpha$, and estimated $\hat{p}$ to obtain an estimate for $n_\alpha$. Note that since $p$ is unknown this is only an estimate, but asymptotically (as $n$ gets larger) it should be accurate.