Reproducibility of the two-sample Kolmogorov–Smirnov test

I am using the two-sample Kolmogorov–Smirnov test to check if two datasets have the same underlying distribution. I can regenerate those two datasets as many times as I want. When I apply the test, however, the computed p-values differ quite dramatically from each other. I am wondering what could be the reason and how one could go about it.

To give an example, here are the p-values that the test yields (I am using MATLAB’s kstest2) when I apply it to 10 pairs of datasets with 10000 samples each (a significance level of 0.05 is assumed):

8.93e-02
1.10e-01
2.41e-05 (reject)
8.52e-01
3.78e-03 (reject)
2.22e-01
2.86e-04 (reject)
3.85e-04 (reject)
1.36e-02 (reject)
9.02e-03 (reject)


As you can see, the results are quite inconsistent. Almost 50/50. I do not know how to interpret these results and would be grateful for any help. In particular, I am interested to know

1. if there is something that I can change in my experimental setup to make the test more consistent/reproducible and

2. if there is a sensible way to summarize multiple p-values into a single one like averaging.

What is also interesting is that drawing only 1000 samples makes many tests pass:

2.35e-01
9.88e-01
5.29e-01
3.94e-01
1.60e-01
1.76e-01
2.63e-03 (reject)
2.82e-01
1.94e-01
2.14e-01


The dependency on the sample size is worrying me as it seems that I can make my test pass by simply reducing the number of samples, which sounds like cheating.

Thank you!

EDIT (more context):

As requested, I would like to provide additional details about what I am actually doing. I have two deterministic algorithms: A and B. The algorithms have random inputs with known distributions. A is the ground truth, and B is an approximation to A. B is supposed to produce statistically the same outputs as A does. I would like to measure how well B approximates A, and my original idea was to apply the Kolmogorov–Smirnov test and monitor the corresponding p-values.

Regards, Ivan

• It sounds like you want to show two things are effectively the same (that they differ by so little that it is of no practical consequence to you). That's not an ordinary goodness of fit hypothesis testing problem. The distributions under A and B will not be 100% identical (obviously not -- they can't be 100% identical, since B is an approximation -- you don't need a test to tell you this, you knew it already). So instead answer a question you don't already know the answer to. – Glen_b Feb 6 '15 at 16:45
• @Glen_b, yes, you are right that I already know that they are not the same. I thought I could use the Kolmogorov–Smirnov test as a goodness-of-fit test (probably my terminology is a bit off here). I would like to show that the difference is practically negligible, and I was hoping that p-values would suite this purpose well. Do you think there is a more sensible way to achieve this? So, yes, it’s not really about equality but rather about tight closeness. – Ivan Feb 6 '15 at 16:54
• "I would like to show that the difference is practically negligible, and I was hoping that p-values would suite this purpose well." -- you have plenty of company. Depending on what sort of features of the distribution are important to you (in what ways does similarity of distribution matter most?), it may be that some kind of equivalence test might be designed, or perhaps some measure of the difference might be used (and in that case, perhaps a confidence interval for that difference measure may be relevant) --- but that's now quite a different question. – Glen_b Feb 6 '15 at 16:59

p-values are random quantities; they're a function of your (random) samples, so naturally they vary from sample to sample.

[Indeed, for a point null hypothesis, and with a continuous rather than discrete test statistic, then if $$H_0$$ were true, the p-values generated in this manner would be uniform on (0,1).]

As the situation moves further and further from the null in the manner measured by the test statistic, the distribution of p-values skews toward the low end. The p-value is always random (with a long tail to the larger p-values), but it tends to be stochastically smaller.

So you're simply expecting something that won't happen. The p-values will never be particularly consistent in value; they don't tend to concentrate around some underlying "population" value.

What you're seeing is how hypothesis tests work.

Your results - considered all together - already indicate the null is false (point nulls are rarely true, so this is not much of a surprise). If you can take larger samples, your typical p-values will become smaller. If you can't take larger samples but can generate as many hypothesis tests as you wish, you could combine them ... and get an overall p-value smaller than any given positive bound.

From the little bit of information you give in your description, it doesn't sound like a hypothesis test is a good choice for your situation. If you describe your underlying problem in greater detail (what you're trying to achieve before you get to the point of deciding to use hypothesis tests), it may be that alternatives more suited to your needs could be suggested.

Answering the question 'do they differ' is probably pointless (they do, at least a little -- enough data will tell you this). A more useful question is probably nearer to "are they different enough that it matters?"

• Thank you! I really appreciate your answer; it clarifies a lot to me. I never thought of p-values in this way. Since I have this luxury of generating more and more data and applying the test over and over again, do you think I should really do it? Should I find the mean of my p-value and base the final decision on that mean? I am wondering if it’ll be something acceptable to discuss and present in a scientific publication. Do you have any comments on the sample size? Thank you! – Ivan Feb 6 '15 at 16:28
• I have updated the question and added some additional details. – Ivan Feb 6 '15 at 16:41
• No, I don't think you should do it. See my edits. If you do want to combine the information from the tests, you don't do it by averaging -- or rather, you don't then compare the average with a significance level; such an average has it's own distribution, and that average would be a test statistic. [The more usual way to combine p-values for different samples is Fisher's method.] -- but since you can generate as many as you like, rejection is certain. [If you can make the samples as large as you like, again rejection is certain] -- why test a null you already know to be false? – Glen_b Feb 6 '15 at 16:41

(1) If the null hypothesis (of no difference between the distributions of the two data-sets) is true, the p-value should be less than 0.05 in no more than 5% of your simulations. So that it's "almost 50/50" should only be surprising if you were quite sure the null hypothesis was true.

(2) Decreasing the sample size decreases the power of the test—a given value of the test statistic can now more plausibly be viewed as arising by chance under the null hypothesis.

• Thank you for your quick answer. The thing is that I need to draw a conclusion and report a number. If the test flips a coin to decide, how can I be sure that my conclusion is sound and the reported number makes sense? What if couldn’t repeat the experiment and had only one pair of datasets? I don’t know what my next step should be. – Ivan Feb 6 '15 at 16:11