How to use ks test for 2 vectors of scores in python?

Question

I am trying to figure out how to determine whether to reject or not the null hypothesis using ks-test.

In matlab there is a function named kstest2 decides if it should reject $H_0$. I want to use python scipy function ks_2samp but it returns the p-value and the $\alpha$, how can I determine if one should reject $H_0$ given those 2 parameters ?
If I have 2 vectors of scores per day, how can I use Matlab/python kstest2 to see if the are distributed the same?
Is there a clean way to change the scores vector to a continuous distributed vector?

Simply compare the p-value to your significance level. If your p-value is lower than (or equal to) your significance level (your chosen type I error rate), you should reject the null hypothesis. — Glen_b, Commented Oct 2, 2013 at 22:59
@Glen_b do you mean if $\alpha > p-value$ reject $H_0$ ? Does $D statistic$ is the same as $alpha$ — 0x90, Commented Oct 2, 2013 at 23:01
Yes and no. Or more strictly, since $\alpha$ is fixed (chosen by you prior to the test), the $\alpha$ is the thing being compared against, so it's the way around I initially stated. And if they happen to be equal you also reject. So I mean it as I said it in words before: if $\text{p-value}\leq\alpha$ you should reject $H_0$. No, $D$ is NOT $\alpha$. $D$ is something you calculate from the sample, while $\alpha$ is something you would normally choose before you even collect the sample and certainly before you look at it. en.wikipedia.org/wiki/Statistical_significance — Glen_b, Commented Oct 2, 2013 at 23:05

Glen_b · Accepted Answer · 2013-10-02 23:12:03Z

Simply compare the p-value to your desired significance level. If your p-value is less than (or equal to) your significance level (your chosen type I error rate, $\alpha$), you should reject the null hypothesis. (You may need to brush up your understanding of how hypothesis testing works.)
If you mean you want to combine information across many days, it depends on whether the days are going to share a distribution (within the two different groups of things being compared in the test) or not, but one approach that works in either case would be to test the distribution of p-values for uniformity against the alternative that it's typically smaller. That would give an overall test that would apply over many days. However, if you're testing every day, you may want to consider the properties of such a procedure.
No. If you don't have continuous distributions you probably shouldn't be doing a KS test at all; it won't have the usual properties (e.g. type I error rates will be too low, power will be low).

Which test can be used for discrete distributions instead of ks-test? — 0x90, Commented Oct 3, 2013 at 3:49
The search function will find you answers related to this, like this or this. There are other possibilities, but it's really a new question. — Glen_b, Commented Oct 3, 2013 at 4:29

1 Answer 1