We expect a uniform distribution of p-values when the null hypothesis is true, so some of the p-values will be small. It’s just a bit of strange luck that the decrease happened to coincide with an increased sample size.

Unless you have a problem with the software implementation (which SciPy should not give), you should end up with the expected uniform distribution of p-values.

import numpy as np
from scipy.stats import norm, ks_2samp
np.random.seed(0)
mylist = []
for n in range(10, 1000000, 100):
   x = norm(0, 4).rvs(n)
   y = norm(0, 4).rvs(n)
   mylist.append(ks_2samp(x, y).pvalue) #not sure abo the exact syntax

plt.hist(mylist) plt.show() plt.close()

You also can try a different seed and see if you still get that same pattern. I often pick the year.

import numpy as np
from scipy.stats import norm, ks_2samp
np.random.seed(2022)
for n in [10, 100, 1000, 10000, 100000, 1000000]:
   x = norm(0, 4).rvs(n)
   y = norm(0, 4).rvs(n)
   print(ks_2samp(x, y))

We expect a uniform distribution of p-values when the null hypothesis is true, so some of the p-values will be small. It’s just a bit of strange luck that the decrease happened to coincide with an increased sample size.

Unless you have a problem with the software implementation (which SciPy should not give), you should end up with the expected uniform distribution of p-values.

import numpy as np
from scipy.stats import norm, ks_2samp
np.random.seed(0)
mylist = []
for n in range(10, 1000000, 100):
   x = norm(0, 4).rvs(n)
   y = norm(0, 4).rvs(n)
   mylist.append(ks_2samp(x, y).pvalue) #not sure about the exact syntax
plt.hist(mylist)
plt.show()
plt.close()

You also can try a different seed and see if you still get that same pattern. I often pick the year.

import numpy as np
from scipy.stats import norm, ks_2samp
np.random.seed(2022)
for n in [10, 100, 1000, 10000, 100000, 1000000]:
   x = norm(0, 4).rvs(n)
   y = norm(0, 4).rvs(n)
   print(ks_2samp(x, y))