I have read these two questions Why are p-values uniformly distributed under the null hypothesis? and Understanding scipy Kolmogorov-Smirnov test

And this inspired me the following experiment.

I consider a number (100) of random samples of 10000 numbers each, drawn from the uniform distribution:

import numpy as np
from scipy.stats import kstest

np.random.seed(1)
data = np.random.rand(1e6).reshape(100, -1) # Retrieve 100 samples of 10000 random numbers
pvals = np.array([kstest(data[i, :], 'uniform')[1] for i in range(100)]) # Use KS test to determine the p-value that they are drawn from a uniform distribution

The p-values should be distributed uniformly between 0 and 1, because the null hypothesis that each sample is drawn from the uniform distribution is true. In the code, pvals contains the p-values, and should be distributed uniformly between 0 and 1. How do I test that they are distributed uniformly? Well, with another KS test on the p-values themselves. Indeed:

kstest(pvals, 'uniform') # gives (0.066826050153764194, 0.78391523133790764)

My question is: how bad should a sample fail its individual KS test, for the p-value distribution not to be uniform? That is, to cause the second KS test to fail as well? Let's inject a couple of failed tests, and see the results:

for i in range(100):
    pvals[i] = 1.e-1000000000000000000
    print(i+1, kstest(pvals, 'uniform'))

1 (0.076826050153764203, 0.58422275090933029)
2 (0.076826050153764203, 0.58422275090933029)
3 (0.086826050153764212, 0.41822788102030262)
4 (0.08849630728801916, 0.39396788117495984)
5 (0.098496307288019169, 0.26906188301811063)
6 (0.10849630728801915, 0.1764729585550886)
7 (0.11849630728801916, 0.11114480850529129)
8 (0.12849630728801917, 0.067209484059870706)
9 (0.13849630728801915, 0.039015977679199176)
10 (0.14849630728801916, 0.021740018916014403)
11 (0.15849630728801917, 0.011625520543988133)
[...]

This tells me that for the second KS test to acknowledge that the p-value distribution is not uniform with 0.99 confidence, I must inject 11 failed tests, each as bad as p-value = 1.e-1000000000000000000.

Intuitively however I expected that the probability of getting one out of 100 p-values less than or equal to 1.e-1000000000000000000 would be given by:

$\dfrac{100!}{1! 99!} p (1-p)^{99}$

Where $p = 10^{-1000000000000000000}$. This expression is of the order of 1.e-999999999999999998, or in layman terms (like I am) rather unlikely. Where is my intuition wrong?

I have read these two questions Why are p-values uniformly distributed under the null hypothesis? and Understanding scipy Kolmogorov-Smirnov test

And this inspired me the following experiment.

I consider a number (100) of random samples of 10000 numbers each, drawn from the uniform distribution:

import numpy as np
from scipy.stats import kstest

np.random.seed(1)
data = np.random.rand(1e6).reshape(100, -1) # Retrieve 100 samples of 10000 random numbers
pvals = np.array([kstest(data[i, :], 'uniform')[1] for i in range(100)]) # Use KS test to determine the p-value that they are drawn from a uniform distribution

The p-values should be distributed uniformly between 0 and 1, because the null hypothesis that each sample is drawn from the uniform distribution is true. In the code, pvals contains the p-values, and should be distributed uniformly between 0 and 1. How do I test that they are distributed uniformly? Well, with another KS test on the p-values themselves. Indeed:

kstest(pvals, 'uniform') # gives (0.066826050153764194, 0.78391523133790764)

My question is: how bad should a sample fail its individual KS test, for the p-value distribution not to be uniform? That is, to cause the second KS test to fail as well? Let's inject a couple of failed tests, and see the results:

for i in range(100):
    pvals[i] = 1.e-1000000000000000000
    print(i+1, kstest(pvals, 'uniform'))

1 (0.076826050153764203, 0.58422275090933029)
2 (0.076826050153764203, 0.58422275090933029)
3 (0.086826050153764212, 0.41822788102030262)
4 (0.08849630728801916, 0.39396788117495984)
5 (0.098496307288019169, 0.26906188301811063)
6 (0.10849630728801915, 0.1764729585550886)
7 (0.11849630728801916, 0.11114480850529129)
8 (0.12849630728801917, 0.067209484059870706)
9 (0.13849630728801915, 0.039015977679199176)
10 (0.14849630728801916, 0.021740018916014403)
11 (0.15849630728801917, 0.011625520543988133)
[...]

This tells me that for the second KS test to acknowledge that the p-value distribution is not uniform with 0.99 confidence, I must inject 11 failed tests, each as bad as p-value = 1.e-1000000000000000000.

Intuitively however I expected that the probability of getting one out of 100 p-values less than or equal to 1.e-1000000000000000000 would be given by:

$\dfrac{100!}{1! 99!} p (1-p)^{99}$

Where $p = 10^{-1000000000000000000}$. This expression is of the order of 1.e-999999999999999998, or in layman terms (like I am) rather unlikely. Where is my intuition wrong?

I have read these two questions Why are p-values uniformly distributed under the null hypothesis? and Understanding scipy Kolmogorov-Smirnov test

And this inspired me the following experiment.

I consider a number (100) of random samples of 10000 numbers each, drawn from the uniform distribution:

import numpy as np
from scipy.stats import kstest

np.random.seed(1)
data = np.random.rand(1e6).reshape(100, -1) # Retrieve 100 samples of 10000 random numbers
pvals = np.array([kstest(data[i, :], 'uniform')[1] for i in range(100)]) # Use KS test to determine the p-value that they are drawn from a uniform distribution

The p-values should be distributed uniformly between 0 and 1, because the null hypothesis that each sample is drawn from the uniform distribution is true. In the code, pvals contains the p-values, and should be distributed uniformly between 0 and 1. How do I test that they are distributed uniformly? Well, with another KS test on the p-values themselves. Indeed:

kstest(pvals, 'uniform') # gives (0.066826050153764194, 0.78391523133790764)

My question is: how bad should a sample fail its individual KS test, for the p-value distribution not to be uniform? That is, to cause the second KS test to fail as well? Let's inject a couple of failed tests, and see the results:

for i in range(100):
    pvals[i] = 1.e-1000000000000000000
    print(i+1, kstest(pvals, 'uniform'))

1 (0.076826050153764203, 0.58422275090933029)
2 (0.076826050153764203, 0.58422275090933029)
3 (0.086826050153764212, 0.41822788102030262)
4 (0.08849630728801916, 0.39396788117495984)
5 (0.098496307288019169, 0.26906188301811063)
6 (0.10849630728801915, 0.1764729585550886)
7 (0.11849630728801916, 0.11114480850529129)
8 (0.12849630728801917, 0.067209484059870706)
9 (0.13849630728801915, 0.039015977679199176)
10 (0.14849630728801916, 0.021740018916014403)
11 (0.15849630728801917, 0.011625520543988133)
[...]

This tells me that for the second KS test to acknowledge that the p-value distribution is not uniform with 0.99 confidence, I must inject 11 failed tests, each as bad as p-value = 1.e-1000000000000000000.

Intuitively however I expected that the probability of getting one out of 100 p-values less than or equal to 1.e-1000000000000000000 would be given by:

$\dfrac{100!}{1! 99!} p (1-p)^{99}$

Where $p = 10^{-1000000000000000000}$, Which is of the order of 1.e-999999999999999998, or in layman terms (like I am) rather unlikely. Where is my intuition wrong?

I have read these two questions Why are p-values uniformly distributed under the null hypothesis? and Understanding scipy Kolmogorov-Smirnov test

And this inspired me the following experiment.

I consider a number (100) of random samples of 10000 numbers each, drawn from the uniform distribution:

import numpy as np
from scipy.stats import kstest

np.random.seed(1)
data = np.random.rand(1e6).reshape(100, -1) # Retrieve 100 samples of 10000 random numbers
pvals = np.array([kstest(data[i, :], 'uniform')[1] for i in range(100)]) # Use KS test to determine the p-value that they are drawn from a uniform distribution

The p-values should be distributed uniformly between 0 and 1, because the null hypothesis that each sample is drawn from the uniform distribution is true. In the code, pvals contains the p-values, and should be distributed uniformly between 0 and 1. How do I test that they are distributed uniformly? Well, with another KS test on the p-values themselves. Indeed:

kstest(pvals, 'uniform') # gives (0.066826050153764194, 0.78391523133790764)

My question is: how bad should a sample fail its individual KS test, for the p-value distribution not to be uniform? That is, to cause the second KS test to fail as well? Let's inject a couple of failed tests, and see the results:

for i in range(100):
    pvals[i] = 1.e-1000000000000000000
    print(i+1, kstest(pvals, 'uniform'))

1 (0.076826050153764203, 0.58422275090933029)
2 (0.076826050153764203, 0.58422275090933029)
3 (0.086826050153764212, 0.41822788102030262)
4 (0.08849630728801916, 0.39396788117495984)
5 (0.098496307288019169, 0.26906188301811063)
6 (0.10849630728801915, 0.1764729585550886)
7 (0.11849630728801916, 0.11114480850529129)
8 (0.12849630728801917, 0.067209484059870706)
9 (0.13849630728801915, 0.039015977679199176)
10 (0.14849630728801916, 0.021740018916014403)
11 (0.15849630728801917, 0.011625520543988133)
[...]

This tells me that for the second KS test to acknowledge that the p-value distribution is not uniform with 0.99 confidence, I must inject 11 failed tests, each as bad as p-value = 1.e-1000000000000000000.

Intuitively however I expected that the probability of getting one out of 100 p-values less than or equal to 1.e-1000000000000000000 would be given by:

$\dfrac{100!}{1! 99!} p (1-p)^{99}$

Where $p = 10^{-1000000000000000000}$. This expression is of the order of 1.e-999999999999999998, or in layman terms (like I am) rather unlikely. Where is my intuition wrong?