Why is p-value produced by the K-S test decreasing with increased sample size even though samples are from the same distribution?

Question

I have two samples that I know a priori were drawn from the same distribution. When I increase the size of the samples, the K-S statistic dips significantly (expected), but so does the p-value, sometimes. It actually exhibits a kind of erratic behavior:

import numpy as np
from scipy.stats import norm, ks_2samp
np.random.seed(0)
for n in [10, 100, 1000, 10000, 100000, 1000000]:
   x = norm(0, 4).rvs(n)
   y = norm(0, 4).rvs(n)
   print(ks_2samp(x, y))

result:

KstestResult(statistic=0.3, pvalue=0.7869297884777761)
KstestResult(statistic=0.09, pvalue=0.8154147124661313)
KstestResult(statistic=0.031, pvalue=0.7228251828701066)
KstestResult(statistic=0.0239, pvalue=0.0066097390500499155)
KstestResult(statistic=0.002950000000000008, pvalue=0.7757569319761763)
KstestResult(statistic=0.0011920000000000819, pvalue=0.4758505446766491)

why did the p-value drop between 2 and 3, and 3 and 4? and in the subsequent iteration, it surges up to 0.7, then drops again. why is that?

More generally, do I even need to look at p-value when interpreting the result of the K-S test? it sounds from the wikipedia article that the null hypothesis can be rejected based on the statistic alone.

Answer 1

We expect a uniform distribution of p-values when the null hypothesis is true, so some of the p-values will be small. It’s just a bit of strange luck that the decrease happened to coincide with an increased sample size.

Unless you have a problem with the software implementation (which SciPy should not give), you should end up with the expected uniform distribution of p-values.

import numpy as np
from scipy.stats import norm, ks_2samp
np.random.seed(0)
mylist = []
for n in range(10, 1000000, 100):
   x = norm(0, 4).rvs(n)
   y = norm(0, 4).rvs(n)
   mylist.append(ks_2samp(x, y).pvalue) #not sure about the exact syntax
plt.hist(mylist)
plt.show()
plt.close()

You also can try a different seed and see if you still get that same pattern. I often pick the year.

import numpy as np
from scipy.stats import norm, ks_2samp
np.random.seed(2022)
for n in [10, 100, 1000, 10000, 100000, 1000000]:
   x = norm(0, 4).rvs(n)
   y = norm(0, 4).rvs(n)
   print(ks_2samp(x, y))