Suppose I have a random number generator and I want to check with a Chi Square Test whether its pdf is uniform or no. I can write a script that does that and I will run it several times. To my surprise, I get completely different results each times. Sometimes I will get a p-value of 0.3, sometimes 0.987 and other times 0.003. Which is the number I should take? Should I try to get an average of the p-values I get? How do I decide if this generator passes the test or not?
So what I try next is using a random number generator that I already "know" to be uniform and I run the test on that. And I keep getting absolutely varying results. Even if I increase the number of samples, it keeps varying a lot! From what I understad, the probability if seeing a very low p-value when drawing random samples from a uniform distribution should be low, and the probability of seeing a high p-value should be high. But this doesn't seem to happen.
How should I interpret the results I get from this test?
This is the Python script I am using:
import numpy as np
from scipy.stats import chisquare
bins=256
x = np.random.randint(bins, size=bins*100)
h = [];
for i in range(bins):
h.append(0);
for n in range(0, len(x)):
h[x[n]] += 1;
print(chisquare(h))
and this are the results I get:
Power_divergenceResult(statistic=303.19999999999999, pvalue=0.020572599306529871)
Power_divergenceResult(statistic=211.06, pvalue=0.97933788750272888)
Power_divergenceResult(statistic=289.66000000000003, pvalue=0.066874498546635575)
Power_divergenceResult(statistic=275.63999999999999, pvalue=0.17885688588645363)
Power_divergenceResult(statistic=257.86000000000001, pvalue=0.43814613213884313)
Power_divergenceResult(statistic=217.07999999999998, pvalue=0.95911527563656596)
Even more, I did a histogram of the p-values I got and it looks pretty uniform. If I know the samples I have are drawn from a uniform distribution, shouldn't I get most of the times a high p-value?
