Kolmogorov-Smirnov p-value and alpha value in python

I am having trouble understanding the scipy.stats documentation against the backdrop of a book that explains the KS-Test.

The documentation (https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html) shows that the function gives us a p-Value which shows us wether we can reject the null-hypothesis if the two samples are from the same distribution or not.

How does this compare to the alpha value explained here: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov%E2%80%93Smirnov_test

For my two sets of data, which look like this:

I get this result:

stats.ks_2samp(original, synth)
Out[366]: KstestResult(statistic=0.05868544600938967, pvalue=3.5569000817378716e-13)


That makes sense to me, since the distributions are fairly similar. Since I have 8760 samples in each set however, my critical D value is 0.0205, so the KstestResult value is bigger than that. Knowing that the test gets harsher for big sample sizes this also seems legit, since there IS a difference in the distributions, even if it is not that big.

But what exactly does the very small pvalue tell me?

Your sample size is huge, so standard errors on the cdf are tiny. Consequently, you can easily tell that the data are not exactly from the distribution. (Goodness of fit tests rarely answer a useful question. I expect that a goodness of fit test is not what you need here either.)

The p-value is the probability of seeing a test statistic at least as large as 0.05868, if the true cdf of the population the same was drawn from was the hypothesized one.

The significance level $$\alpha$$ is the cutoff you would use for the p-value (if you did it that way instead of by comparing the test statistic to the critical value); if $$p\leq \alpha$$ then $$D\geq D_\text{crit}$$.

Note that $$\alpha$$ is something you choose prior to testing (properly, prior to collecting the data), while $$p$$ is a property of your sample.

https://en.wikipedia.org/wiki/P-value

https://en.wikipedia.org/wiki/Statistical_significance#Related_concepts

Being a bit loose with the phrasing, I would phrase it as follows.

1. The distributions are pretty similar.

2. The distributions are, however, not the same, and you’re very confident about that.

Being more formal about interpreting the p-value, if the two distributions are the same, you’ve observed an unusual result.

• thanks for your answer! I am not sure I understand it fully. So if p is very low and the K-S-Test Value is higher than the critical c(alpha) value, then I am very confident that the distributions are not the same? But wouldn't this be different from what they write in the docs in scipy? Apr 5, 2022 at 12:35
• @Stefan44 What particular comment in the documentation seems inconsistent with what I wrote?
– Dave
Apr 5, 2022 at 12:39
• where they write about rejecting the null because of low p values. My p value is low, but does the null change depending on what my relation with the critical value is? Apr 5, 2022 at 12:55
• What do you mean by the relation with the critical value?
– Dave
Apr 5, 2022 at 13:08
• i just realized that the docs are full of mistakes, thats where a lot of my confusion comes from actually. But one question remains: what is the difference between alpha and p in here? Apr 5, 2022 at 14:55