# How far from "even" is (or is not) significant?

If a six-sided die is rolled 600 times, an even distribution would be 100 rolls for each number. If I actually rolled 600 times and tallied the results...

1. How far away from 100 would a count need to be before I thought "there's something fishy with that die"?

2. How far away from 100 would it have to be to be "certain" something was wrong?

And...

1. If I wanted to test a die, how many times would I need to roll it to be reasonably sure it was either truly random or weighted somehow? 100? 600? 1000? 1000000? Infinity times? Is there a point of diminishing returns where rolling x times provides 99.9% certainty but to increase my certainty I'd have to roll it 2x or 10x more times so it is't worth it? (Unless it is?)

2. What would "reasonably sure" mean? Are there different values of "reasonably sure" for different applications? Would a D&D player, a dice manufacturer, a casino, or a (hypothetical) high-value encryption algorithm that relies on absolute randomness have different values for "reasonably sure"?

• These questions are primarily psychological in nature. For ways to formulate them statistically, please search our site for threads about p-values, power, sample size, and hypothesis testing. A common statistical method applied in this case is the chi square test, something else you can read a lot about here on CV.
– whuber
Nov 6, 2020 at 18:38
• I'm not exactly interested in the psychology. I'm interested in what constitutes the difference between statistical significance and typical random distribution. I don't know enough vocabulary to ask it a better way. Nov 6, 2020 at 19:26
• It's a large topic: that's why I offered the basic search words earlier. If a generous reader were to focus on your question #1 and offer a standard interpretation, they might be able to achieve a reasonably brief reply; but otherwise you're trying to ask for too much at once.
– whuber
Nov 6, 2020 at 19:36

Judging unfairness. A common way to judge whether faces on a die are equally likely is to use a chi-squared goodness-of-fit test.

Suppose you roll a fair die 600 times and count the numbers of 1s, 2s, ..., 6s observed. We can simulate this by using R:

set.seed(116)  # for reproducibility
d = sample(1:6, 600, rep=T)
x = tabulate(d);  x
[1] 103 116  88  98  91 104


As you say, the expected counts are $$E_i = 100, i = 1,\dots,6.$$

The chi-squared statistic is the sum

$$Q = \sum_{i=1}^6 \frac{(X_i-E_i)^2}{E_i} = \frac{(103 - 100)^2}{100} + \cdots + \frac{(104-100)^2}{100} = 5.1.$$

sum((x - 100)^2/100)
[1] 5.1


If the die is 'behaving' fairly, then the $$X_i$$s will tend to be relatively near to the expected value $$100$$ and the terms of $$Q$$ will be relatively small. If the die is biased, then $$Q$$ will tend to be large. The question is how large can $$Q$$ be before we begin to suspect the die is unfair.

Under the null hypothesis that the die is fair, $$Q \stackrel{aprx}{\sim} \mathsf{Chisq}(\nu = 6-1=5),$$ the chi-squared distribution with five degrees of freedom. If we cut 5% of the probability from the upper tail of $$\mathsf{Chisq}(5),$$ we get the 5% critical value $$c = 11.07.$$

So if $$Q \ge c = 11.07$$ we say that is evidence at the 5% level of significance that the die is not fair. We conclude that our data above give no evidence of unfairness.

qchisq(.95, 5)
[1] 11.0705


Thus, traditionally, we do not look at the behavior of any one of the six counts to decide whether a die is fair, but we look at the overall 'profile' of all six counts as measured by the chi-squared GOF statistic $$Q.$$

Required sample size. Another part of your question has to do with the number of rolls of the die it takes in order to detect unfairness. Suppose that the die is unfair in such a way that the respective faces have probabilities $$(2/18, 3/18, 3/18,$$ $$3/18, 3/18, 4/18).$$ [Maybe someone put a small lead weight into the die on side 1, so the 1 occurs too seldom and 6 too often.] Are $$n = 600$$ rolls of the die enough to have a good chance of detecting this degree of unfairness?

Let's look at two sessions of 600 rolls of such an unfair die as simulated in R:

set.seed(1234)
d = sample(1:6, 600, rep=T, p=c(2,3,3,3,3,4)/18)
x = tabulate(d); x
[1]  72 105 103  93 106 121
sum((x-100)^2/100)
[1] 13.44

d = sample(1:6, 600, rep=T, p=c(2,3,3,3,3,4)/18)
x = tabulate(d); x
[1]  74  79 107  96 101 143
sum((x-100)^2/100)
[1] 30.32


It happens that both sessions gave values of $$Q$$ that are large enough to detect unfairness.

In R, the procedure chisq.test automates the testing, giving a P-value smaller than 0.05 if the the null hypothesis should be rejected. For the first example above, we have x = c(72, 105, 103, 93, 106, 121). [Unless the contrary is stated, chisq.test assumes the null hypothesis is that all faces are equally likely.]

chisq.test(x)

Chi-squared test for given probabilities

data:  x
X-squared = 13.44, df = 5, p-value = 0.01959


Thus, it is possible to do a simulation for $$m = 100,000$$ 600-roll experiments with a specific kind of unfair die in order to see how often unfairness is detected. The answer is: almost always. So $$n =600$$ is plenty of rolls to detect this level of unfairness.

set.seed(2020)
unf = c(2,3,3,3,3,5)/18
pv = replicate(10^5, chisq.test(tabulate(
sample(1:6,600,rep=T,p=unf)))$p.val) mean(pv <= .05) [1] 0.99986  However, if I try to get by with only $$n = 150$$ rolls of this biased die I will detect biasedness only about 3/4 of the time. set.seed(2020) unf = c(2,3,3,3,3,5)/18 pv = replicate(10^5, chisq.test(tabulate( sample(1:6,150,rep=T,p=unf)))$p.val)
mean(pv <= .05)
[1] 0.74368


And with only $$n = 60$$ rolls, we would detect unfairness only about $$1/3$$ of the time [simulation not shown].

We say that the power of the test is about $$.99, .75,$$ and $$.33$$ for $$n = 600, 150,$$ and $$60,$$ respectively. There are formulas for determining the power of the chi-squared GOF test for various scenarios of unfairness. [If you are interested in the technical details you can look at this Q&A or try googling power chi-squared test.]