Kolmogorov-Smirnov p-value and alpha value in python I am having trouble understanding the scipy.stats documentation against the backdrop of a book that explains the KS-Test.
The documentation (https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html) shows that the function gives us a p-Value which shows us wether we can reject the null-hypothesis if the two samples are from the same distribution or not.
How does this compare to the alpha value explained here: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov%E2%80%93Smirnov_test
For my two sets of data, which look like this:

I get this result:
stats.ks_2samp(original, synth)
Out[366]: KstestResult(statistic=0.05868544600938967, pvalue=3.5569000817378716e-13)

That makes sense to me, since the distributions are fairly similar.
Since I have 8760 samples in each set however, my critical D value is 0.0205, so the KstestResult value is bigger than that. Knowing that the test gets harsher for big sample sizes this also seems legit, since there IS a difference in the distributions, even if it is not that big.
But what exactly does the very small pvalue tell me?
 A: Your sample size is huge, so standard errors on the cdf are tiny. Consequently, you can easily tell that the data are not exactly from the distribution.  (Goodness of fit tests rarely answer a useful question. I expect that a goodness of fit test is not what you need here either.)
The p-value is the probability of seeing a test statistic at least as large as 0.05868, if the true cdf of the population the same was drawn from was the hypothesized one.
The significance level $\alpha$ is the cutoff you would use for the p-value (if you did it that way instead of by comparing the test statistic to the critical value); if $p\leq \alpha$ then $D\geq D_\text{crit}$.
Note that $\alpha$ is something you choose prior to testing (properly, prior to collecting the data), while $p$ is a property of your sample.
https://en.wikipedia.org/wiki/P-value
https://en.wikipedia.org/wiki/Statistical_significance#Related_concepts
A: Being a bit loose with the phrasing, I would phrase it as follows.

*

*The distributions are pretty similar.


*The distributions are, however, not the same, and you’re very confident about that.
Being more formal about interpreting the p-value, if the two distributions are the same, you’ve observed an unusual result.
