I have read these two questions https://stats.stackexchange.com/questions/10613/why-are-p-values-uniformly-distributed-under-the-null-hypothesis and https://stats.stackexchange.com/questions/113464/understanding-scipy-kolmogorov-smirnov-test And this inspired me the following experiment. I consider a number (100) of random samples of 10000 numbers each, drawn from the uniform distribution: import numpy as np from scipy.stats import kstest np.random.seed(1) data = np.random.rand(1e6).reshape(100, -1) # Retrieve 100 samples of 10000 random numbers pvals = np.array([kstest(data[i, :], 'uniform')[1] for i in range(100)]) # Use KS test to determine the p-value that they are drawn from a uniform distribution The p-values should be distributed uniformly between 0 and 1, because the null hypothesis that each sample is drawn from the uniform distribution is true. In the code, `pvals` contains the p-values, and should be distributed uniformly between 0 and 1. How do I test that they are distributed uniformly? Well, with *another* KS test on the p-values themselves. Indeed: kstest(pvals, 'uniform') # gives (0.066826050153764194, 0.78391523133790764) My question is: how bad should a sample fail its individual KS test, for the p-value distribution not to be uniform? That is, to cause the *second* KS test to fail as well? Let's inject a couple of failed tests, and see the results: for i in range(100): pvals[i] = 1.e-1000000000000000000 print(i+1, kstest(pvals, 'uniform')) 1 (0.076826050153764203, 0.58422275090933029) 2 (0.076826050153764203, 0.58422275090933029) 3 (0.086826050153764212, 0.41822788102030262) 4 (0.08849630728801916, 0.39396788117495984) 5 (0.098496307288019169, 0.26906188301811063) 6 (0.10849630728801915, 0.1764729585550886) 7 (0.11849630728801916, 0.11114480850529129) 8 (0.12849630728801917, 0.067209484059870706) 9 (0.13849630728801915, 0.039015977679199176) 10 (0.14849630728801916, 0.021740018916014403) 11 (0.15849630728801917, 0.011625520543988133) [...] This tells me that for the second KS test to acknowledge that the p-value distribution is not uniform with 0.99 confidence, I must inject 11 failed tests, each as bad as p-value = 1.e-1000000000000000000. Intuitively however I expected that the probability of getting one out of 100 p-values less than or equal to 1.e-1000000000000000000 would be given by: $\dfrac{100!}{1! 99!} p (1-p)^{99}$ Where $p = 10^{-1000000000000000000}$. This expression is of the order of 1.e-999999999999999998, or in layman terms (like I am) rather unlikely. Where is my intuition wrong?