enter image description here

Drawing 220 times with replacement from a lottery machine with 12 balls numbered 1 to 12 results in the following distribution:

ball no.     1  2  3  4  5  6  7  8  9 10 11 12
frequency   23 18 21 15 24 17 20 16 21 13 19 13

plot of distribution

As you can see, the odd numbers are drawn more frequently than the even numbers and the lower numbers more frequently than the higher numbers.

Is that an artifact of the small sample size, or is there some plausible explanation for this unequal distribution?


  1. These are real numbers drawn from a real machine in a real lottery.
  2. The machine that was used is not the one pictured but similar to it, i.e. a spherical cage manually operated (not a glass sphere and not an automaton).
  3. I investigated this lottery because in the German national lottery ("6 aus 49"), where 6 numbers are drawn from an urn containing 49 numbers, the numbers haven't been drawn with the same frequency in the 5000 or so drawings since 1955 (and several changes of the lottery machine), with 6 being the most frequent number (drawn 611 times) and 13 the most infrequent (492 times). More detailed and complex (descriptive) statistics of the German national lottery can be found here (in German). I'm interested in the present lottery, because it is by a non-profit organization that I trust and support and was curious if their machinery or process might produce an unintenional bias. I don't suspect fraud.
  • 3
    $\begingroup$ The only reasonable possible source of bias would be in the balls. I would quickly check if the odd balls had a slightly different weight or rolled differently, but most likely it's nothing. $\endgroup$ – eps Apr 12 at 13:56
  • 2
    $\begingroup$ A basic analysis of the German lottery data might give $\chi^2=55.719$ with $48$ degrees of freedom, and thus a $p$-value $0.2071$, which does not suggest that is anything more than usual random fluctuation. $\endgroup$ – Henry Apr 13 at 11:06
  • 1
    $\begingroup$ Since the question also requested plausible explanations, and thinking along the same lines as @eps I did quickly look at the area that would be covered by ink on the various balls. Using "Lucida Sans" as a generic test font, odd numbers were least inked (and of course double-digits were heavier, 11 least so). Therefore I must conclude that more ink makes a ball less likely to be drawn. I kid, but for for anyone interested in playing with the data, inked pixel counts for 1-12 (at 100px) were 787, 1217, 1231, 1327, 1223, 1575, 1002, 1727, 1559, 2336, 1574, 2004. [I did NOT underline eg 6/9] $\endgroup$ – A C Apr 14 at 5:07

You mentioned that odd-even pattern, so let's investigate that.

Category Observed Expected # Expected
odd 92 110 50%
even 128 110 50%

And test with just these two categories....

Chi squared equals 5.891 with 1 degrees of freedom.
The two-tailed P value equals 0.0152

That would generally be considered significant This kind of result only happens fifteen times out of a thousand. Does that mean that we have a publishable result?

The binomial test (more exact when there are two categories) gives p = 0.0062, also significant.

There are several problems here. Firstly we chose to focus on one aspect of the data after we had collected it. We ignored others. For example, we could have tested for multiples of 3, or primes/non primes or ... many other choices are possible. Nearly all those other choices find "not significant". This is p-hacking the green jellybean fallacy, illustrated by xkcd. By subdividing or grouping data it is often possible to find a result that is significant. The proper response is to investigate further. If this is a real effect, it should become a stronger result with more data. If it just "green jellybeans" then it won't be repeatable.

And (although not the same experiment) the German 6-aus-49 lottery has no odd-even pattern.

Secondly we have a strong "prior": We inspect the machine and the balls and don't notice anything that seems amiss. We know from our general experience of such machines that they are usually quite fair. What would you estimate the probablity of the machine, or the operator, being substantially unfair, prior to the experiment? Clearly there is some judgement here, but suppose we say 0.001 (one in a thousand chance) Now, given this result, how likely is it that the machine is substantially unfair. Well, this will take you into Bayesian statistics. We would need to quantify "substantially unfair", but if an unfair machine will always give results like this [P(X²>5.891 | unfair)=1] then the probablity that the machine is unfair, given that we observe X²>5.891 is just 0.001/0.0152 =0.06. So we update our estimate of the machine being unfair from 0.001 to 0.06. We still think it rather unlikely that the machine is unfair.

  • 10
    $\begingroup$ Good answer. I'd clarify that the real problem isn't that we're looking at only one aspect of the data (the even/odd issue), but that we decided which aspect to focus on after seeing the data. If we had decided to test even/odd before looking at the data, it would be a more meaningful result. $\endgroup$ – Nuclear Hoagie Apr 12 at 12:10
  • 1
    $\begingroup$ @NuclearHoagie And this also means that if we collect more data, and we still see this pattern in the new data, we can assume it's probably a real pattern. $\endgroup$ – wizzwizz4 Apr 12 at 13:33
  • $\begingroup$ The binomial test gives p = 0.00621804354. $\endgroup$ – Acccumulation Apr 12 at 16:20

To determine whether the results seem to indicate some shenanigans were afoot, we can test it!

To begin, we need to specify what our null hypothesis is. I'll take the moment here to stress (as subsequent answers have pointed out in more detail) the importance of forming hypotheses before seeing the data we will use for testing this hypothesis. For example you observed that odd and lower-valued numbers appear more often, but our hypothesis should be formed prior to seeing the data. As such, it would be wrong to look at the data, note that odd numbers appear more often in the data, and then test that hypothesis based on the same data (as @Him points out, it's certainly reasonable to collect new data to test this hypothesis).

Based on your writing, it seems like past data has made you believe that inconsistencies may be occurring, so a natural test would be whether the draws are from a (discrete) uniform distribution (the null hypothesis), or whether the draws are from some other distribution which would thus indicating bias towards certain numbers (the alternative hypothesis).

A simple test of this hypothesis is a chi square test. Under the null hypothesis, each number has an equal chance of being drawn (i.e. for each draw, there is a $1/12$ chance that any given number is drawn -- subsequent answers have also shown how a chi square test can be used with different hypothesis such as odd/even difference). Under our null, we can calculate how many draws each number should have had by multiplying $1/12$ by the total number of draws for each number.

The test essentially measures by how much the observed draws differ from the expected number of draws under the null. Using your observed numbers and performing this test (see code below), we find that the p-value is $p = 0.71$, and thus we fail to reject the null that the draws came from a fair process. Thus, we cannot conclude that the draws came from a biased sampling process.

>obs = c(23,18,21,15,24,17,20,16,21,13,19,13) 
>p = rep(1/length(obs),length(obs)) 
Chi-squared test for given probabilities
data:  obs X-squared = 8, df = 11, p-value = 0.7133

As for the second part of the question, suppose we did conclude the draws came from unequal probabilities (which we did not). There are tons of plausible explanations for why this may be. To list two:

  1. A hustler with enough skill could probably easily make the process seem fair when it is not, either by cranking the machine in a consistent way that ensures the numbers that started at the bottom end at the bottom after they are done turning the lever, or by simply following the balls while he cranks it.

  2. You say it was from a real lottery, but did you actually observe this process? If not, then it's easy to say these were due to a fair lottery even if they were not.

  • $\begingroup$ In response to OP's updates to his post, I updated my answer to clarify some issues and why I choose the null hypotheses I did (rather than, say, testing odd vs even, for all the excellent reasons pointed out by James K). $\endgroup$ – doubled Apr 12 at 14:08
  • 1
    $\begingroup$ You write "so a natural hypothesis based on past data would be [...]" but what you then write is the alternative hypothesis, not the null hypothesis, since you write "the draws are not from a (discrete) uniform distribution". I think you could make this clearer. Nice answer btw! $\endgroup$ – Vincent Apr 12 at 15:23
  • $\begingroup$ @Vincent yeah you're absolutely right -- tried to clear it up. Hopefully this makes more sense! $\endgroup$ – doubled Apr 12 at 15:37
  • 5
    $\begingroup$ "it would be wrong to look at the data, note that odd numbers appear more often in the data, and then make that our hypothesis" I would like to wholeheartedly disagree with this. What would be wrong is to then attempt to compute a confidence interval or p-value with the "ordinary" interpretations from this same data. Once you've looked at the data, your best course is to form hypotheses based on the data, then establish a hypothesis testing plan, and then gather more data. $\endgroup$ – Him Apr 12 at 23:50
  • 1
    $\begingroup$ @Him Yes, absolutely. What I meant was in the context of hypothesis testing using that same data. I guess it was not fully clear--I'll update my post, thanks. $\endgroup$ – doubled Apr 12 at 23:56

Following @doubled's (+1) chi-squared test, a remaining question is whether 220 draws from the machine are enough to detect an actual small bias. Maybe the odd numbered balls are a bit heavier, lighter, or less round in such a way that they are slightly more likely to be be drawn. Maybe the true probability distribution is $(6,4,6,4, 6,4,6,4, 6,4,6,4)/60.$ What is the the probability a chi-squared test based on 220 draws would detect this bias in favor of odd numbers?

Based on this distribution, one can simulate $m=100\,000$ sessions of 220 draws each, do the chi-squared test each time, and see what fraction of the $m$ sessions rejects the null hypothesis that draws are fair. This gives a good approximation of the power of the chi-squared test to detect the specified degree of unfairness.

To begin, let's look at one such simulated session--which happens not to detect the unfairness (P-value > 5%). [Using R.]

# one session
set.seed(411) # for reproducibility 
pr = c(6,4,6,4, 6,4,6,4, 6,4,6,4)/60
x = sample(1:12, 220, rep=T, p = pr)
TB = tabulate(x);  TB
[1] 23 11 17 20 20 16 19 19 20 17 21 17
[1] 0.898677

Now, by simulating $m$ sessions of $n=220$ draws each, we see that we have just a little less than a 50:50 chance of detecting this level of unfairness. At the end of the simulation run vector pv has $m$ P-values, and mean(pv <= 0.05) gives the proportion of rejections. [The parameter nbins=12 of tabulate forces tabulate to give the correct tally, even if a session lacks some of the higher numbers.]

# 100,000 sessions
pr = c(6,4,6,4, 6,4,6,4, 6,4,6,4)/60
m = 10^5;  pv = numeric(m)
for(i in 1:m) {
 x = sample(1:12, 220, rep=T, p = pr)
 TB = tabulate(x, nbins=12)
 pv[i] = chisq.test(TB)$p.val
mean(pv <= 0.05)
[1] 0.45349

Also, a run of the program with $n = 500$ draws per session [not shown] gives power almost 90%, and a run with $n=650$ gives power just above 95%.

Note: In these simple cases, it is not necessary to do a simulation in order to approximate the power of a chi-squared test of $H_0: \mathrm{Fair}$ against alternative vector pr, using $n$ draws.

The 5% critical value $c=19.6751$ has $P(Q > c|H_0) = 0.05.$ And the 'effect size' is $\lambda = n\sum\frac{(p_{ai} - 1/12)^2}{1/12} = 8.8.$ Then the exact power $0.4602$ is found using the chi-squared distribution with degrees $\nu = 12-1 = 11$ and noncentrality parameter $\lambda.$

c = qchisq(.95, 11);  c
[1] 19.67514
lam = 220*sum((pr-1/12)^2/(1/12)); lam
[1] 8.8
1 - pchisq(c,11,lam)
[1] 0.4602406

By contrast with $n = 650,$ we have $\lambda = 26$ and power $0.9574.$

lam = 650*sum((pr-1/12)^2/(1/12)); lam
[1] 26
1 - pchisq(c,11,lam)
[1] 0.9573635

Perhaps see this Q&A and its references.


As an addition to the other answers, let me offer you a visual way to inspect the differences between the expected and observed frequencies: A (hanging) rootogram, invented by John Tukey (see also Kleiber & Zeileis (2016)). In the figure below, the square roots of the expected counts are displayed as red dots. The square roots of the observed frequencies are hanging as histogram-like bars from these points. Therefore, if the expected and the observed frequencies are similar, the base of the bars all lie near zero. Bars for lower than expected frequencies lie above the zero line and vice versa. This plot makes it easy to see departures in either direction.

In terms of interpretation, recall that for most values, the square root of a count will be less than one unit from its expected value. This is the case for all twelve numbers of your data. Hence, there is little evidence for a systematic departure from uniformity, as the other answers have already explained. The square root is useful here because it's a variance stabilizing function for the Poisson distribution. With $n=220, p=1/12$, the Poisson distribution is a reasonable approximation to the binomial. Finally, for fairly large $\lambda$s, the variance of the square root of a Poisson distribution will be around $1/4$ so the 68-95-99.7 rule suggests most values will lie within $2\times \sqrt{1/4} = 1$ of the mean. One could entertain the arcsine transformation for the binomial, but it virtually makes no difference in this case.


On a separate note, please keep in mind that the usage of a formal hypothesis test could be dubious in such cases, if you only looked at the data because they made you suspicious (this is called HARKing: Hypothesizing after the results are known). In your case, you formed the hypothesis based on other data, which seems fine to me.

  • 1
    $\begingroup$ I looked at the data because numbers in other lotteries don't appear with equal frequency (see note 3, which I added to my question). I formed my hypothesis (that numbers don't appear with equal frequency) before I analysed the data. $\endgroup$ – fluctuating psychosis Apr 12 at 13:07
  • $\begingroup$ @fluctuatingpsychosis Thanks, I edited the section of my answer accordingly. $\endgroup$ – COOLSerdash Apr 12 at 13:27
  • $\begingroup$ @fluctuatingpsychosis That isn't a sufficient hypothesis. Your hypothesis should be “numbers don't appear with equal frequency in [specific way]”; otherwise you have to analyse all possible ways they might appear in not-equal frequency to determine how much evidence there is for your full hypothesis. You changed the hypothesis from “there's something fishy going on” to “this specific thing is the fishy bit”. $\endgroup$ – wizzwizz4 Apr 12 at 13:35
  • $\begingroup$ I don't see why the square root is particularly useful in this case. $\endgroup$ – Acccumulation Apr 12 at 16:23
  • 2
    $\begingroup$ In a chi-squared test the test statistic is $Q = \sum_{i=1}^{12} \frac{(X_i - E)^2}{E},$ where $X_i$ are observed counts out of $n$ draws, and $E = n/12.$ The 12 'contributions' to $Q$ are $C_i = \frac{(X_i - E)^2}{E},$ Sometimes one looks at 'Pearson residuals' $R_i,$ which are 'signed' square roots of $C_i.$ Thus, $Q = \sum_i R_i^2.$ In case $H_0$ is rejected, one can look at largest $|R_i|$ to get an idea which cells contribute most heavily to that decision. $\endgroup$ – BruceET Apr 12 at 19:23

I found that there was a correlation of -0.5109730443013045 between the number drawn and its frequency, and a p-value of 0.090274 for that correlation. For a binomial test for a number being odd or even, I got a p-value of 0.00621804354. When I adjusted the odd numbers for the difference in means (that is, added (even mean-odd mean) to the frequencies of odd numbers), I found a correlation of -0.753098254512706 for the numbers versus frequencies, for a p-value of 0.0047.

As other answers have discussed, this is HARKing (hypothesizing after the results are known). There are lots of different patterns you could have noticed, so given any alpha, the probability that at least one has a p-value less than alpha is significantly greater than the probability for each one individually. Besides the possibility of cherry-picking the hypothesis to fit the data, there is the possibility of cherry-picking the data: if you see lots of lotteries, and only discuss the ones with unusual results, or you stop collecting data from a lottery once that data seems anomalous, and don't continue collecting data to see whether it reverts to the mean, the apparent p-value can misrepresent the true improbability of the results.

0.090274 is a large p-value in general, and in the context of HARKing, it's well within the range of what we could expect by chance. It's enough to draw attention to it, but not enough to come to a solid conclusion. 0.00621804354, on the other hand, is quite small. With it being less that one-eighth of the standard alpha of 0.05, a relevant question to ask is "Are there eight other hypotheses that would be as, or more, noticeable?" It's on the borderline (and keep in mind that you need to account for the bias that the pattern that you actually say feels more salient than ones that you didn't see). When you add in the fact that there are surely more than 161 lotteries, seeing a p-value of 1/161 hardly eliminates the possibility that this is due to just chance. The p-value of 0.0047, or 1/213, for the adjusted correlation is a bit lower, but it's also more contrived, so we need to adjust our "How many hypotheses are there that would be as, or more, noticeable?" number upwards.

So it's enough that it would be understandable to investigate whether there's something causing a bias, but there's no way to know without empirical investigation, and sometimes things happen just by chance. That's the whole point of a lottery, after all. If you were to win a lottery, you probably wouldn't say "The p-value of me winning is less than 0.05, so obviously the lottery's rigged."

  • $\begingroup$ Why stop at 16 digits? 0.5109730443013045 could be so much more precise...! ;-) $\endgroup$ – user2705196 Apr 14 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.