I am using the two-sample Kolmogorov–Smirnov test to check if two datasets have the same underlying distribution. I can regenerate those two datasets as many times as I want. When I apply the test, however, the computed p-values differ quite dramatically from each other. I am wondering what could be the reason and how one could go about it.
To give an example, here are the p-values that the test yields (I am using MATLAB’s
kstest2) when I apply it to 10 pairs of datasets with 10000 samples each (a significance level of 0.05 is assumed):
8.93e-02 1.10e-01 2.41e-05 (reject) 8.52e-01 3.78e-03 (reject) 2.22e-01 2.86e-04 (reject) 3.85e-04 (reject) 1.36e-02 (reject) 9.02e-03 (reject)
As you can see, the results are quite inconsistent. Almost 50/50. I do not know how to interpret these results and would be grateful for any help. In particular, I am interested to know
if there is something that I can change in my experimental setup to make the test more consistent/reproducible and
if there is a sensible way to summarize multiple p-values into a single one like averaging.
What is also interesting is that drawing only 1000 samples makes many tests pass:
2.35e-01 9.88e-01 5.29e-01 3.94e-01 1.60e-01 1.76e-01 2.63e-03 (reject) 2.82e-01 1.94e-01 2.14e-01
The dependency on the sample size is worrying me as it seems that I can make my test pass by simply reducing the number of samples, which sounds like cheating.
EDIT (more context):
As requested, I would like to provide additional details about what I am actually doing. I have two deterministic algorithms:
B. The algorithms have random inputs with known distributions.
A is the ground truth, and
B is an approximation to
B is supposed to produce statistically the same outputs as
A does. I would like to measure how well
A, and my original idea was to apply the Kolmogorov–Smirnov test and monitor the corresponding p-values.