Confused about how to interpret Chi square results

For my local charity I have to organize random draws, each week to give a prize. I just bought a random machine generator, which randomly spits one ball out of 10.

Before using the machine in public I made some tests in private to ensure that the machine is properly random and I am not accused of cheating.

So I made about 1300 random draws and performed a chisquare test.

I got the following chi square value: 18.13

Since there are 10 possible outcomes, there are 9 degrees of freedom.

Here are the values for 9 df:

ν     0.100  0.050   0.025   0.010   0.005   0.001
9   14.6837 16.9190 19.0228 21.6660 23.5893 27.8772


18.13 is greater that 14.6 and 16.9 does it mean that I am 95% confident that the machine is not random and I should change it? Or is 1300 experiments not enough to make a conclusion?

If the machine is random, you can expect $\frac{1300}{10}=130$ draws out of 1300 for any given ball.  (This is well above the recommended minimum expected value of 5 per category, so you should be fine.)

With the observed frequencies for each ball, calculate the squared Pearson residual: $$R_i^2 = \frac{(O_i-E_i)^2}{E_i}$$ (again, for this problem, $E_i = 130$ for all the categories).

The sum of these (squared) residuals is the chi-squared test statistic $\chi^2=\sum R_i^2$, with 9 degrees of freedom (one less than the number of categories).  Last, you need to calculate the $P$-value.  Using R

pchisq(18.13, 9, lower.tail = TRUE)


(where I used the chi-square test statistic you reported in the question).  I get p=.03.  So, we can assess the null hypothesis.

If the machine is random, the observed frequencies should be very close to the expected frequencies. Thus, the null could be $$H_0 : \text{the machine is random}$$ or $$H_0 : p_1 = p_2 = ··· = p_{10} = \frac{1}{10}$$ If you use a conventional significance level of $\alpha=.05$, then you would reject the null (machine is not random).  If you use $\alpha=.02$, then you would fail to reject the null (no reason to think the machine is not random).

The problem here is that your sample size can actually be too large.

• ok thank you, so the interpretation is purely subjective here? Would you reject or accept the null hypothesis personally? I also dont understand when you say that the sample is too large, isnt it better when we have more data points? – user2505650 Apr 22 '18 at 12:01
• For chi-square goodness-of-fit tests, very large samples always lead to p-values that are less than the significance level...so you are always rejecting the null hypothesis (this is referred to as being over-powered). As for the decision for the hypothesis test, that is entirely your call...you pick the $\alpha$ level you feel comfortable with and go with it. – Gregg H Apr 22 '18 at 12:21
• ok thank you, do you recommend any test for this kind of problem? – user2505650 Apr 22 '18 at 12:48
• If I were conducting this type of test, I would run the test with a goal of an expected frequency of ≈15 (ie 150 trials). If I were being extra careful, I would run the same test 100 times to see if about 5 of the tests are flagged as statistically significant. – Gregg H Apr 22 '18 at 14:01