I am trying to figure out how to determine whether to reject or not the null hypothesis using ks-test.

  1. In matlab there is a function named kstest2 decides if it should reject $H_0$. I want to use python scipy function ks_2samp but it returns the p-value and the $\alpha$, how can I determine if one should reject $H_0$ given those 2 parameters ?
  2. If I have 2 vectors of scores per day, how can I use Matlab/python kstest2 to see if the are distributed the same?
  3. Is there a clean way to change the scores vector to a continuous distributed vector?
  • $\begingroup$ Simply compare the p-value to your significance level. If your p-value is lower than (or equal to) your significance level (your chosen type I error rate), you should reject the null hypothesis. $\endgroup$ – Glen_b Oct 2 '13 at 22:59
  • $\begingroup$ @Glen_b do you mean if $\alpha > p-value$ reject $H_0$ ? Does $D statistic$ is the same as $alpha$ $\endgroup$ – 0x90 Oct 2 '13 at 23:01
  • $\begingroup$ Yes and no. Or more strictly, since $\alpha$ is fixed (chosen by you prior to the test), the $\alpha$ is the thing being compared against, so it's the way around I initially stated. And if they happen to be equal you also reject. So I mean it as I said it in words before: if $\text{p-value}\leq\alpha$ you should reject $H_0$. No, $D$ is NOT $\alpha$. $D$ is something you calculate from the sample, while $\alpha$ is something you would normally choose before you even collect the sample and certainly before you look at it. en.wikipedia.org/wiki/Statistical_significance $\endgroup$ – Glen_b Oct 2 '13 at 23:05
  1. Simply compare the p-value to your desired significance level. If your p-value is less than (or equal to) your significance level (your chosen type I error rate, $\alpha$), you should reject the null hypothesis. (You may need to brush up your understanding of how hypothesis testing works.)

  2. If you mean you want to combine information across many days, it depends on whether the days are going to share a distribution (within the two different groups of things being compared in the test) or not, but one approach that works in either case would be to test the distribution of p-values for uniformity against the alternative that it's typically smaller. That would give an overall test that would apply over many days. However, if you're testing every day, you may want to consider the properties of such a procedure.

  3. No. If you don't have continuous distributions you probably shouldn't be doing a KS test at all; it won't have the usual properties (e.g. type I error rates will be too low, power will be low).

  • $\begingroup$ Which test can be used for discrete distributions instead of ks-test? $\endgroup$ – 0x90 Oct 3 '13 at 3:49
  • $\begingroup$ The search function will find you answers related to this, like this or this. There are other possibilities, but it's really a new question. $\endgroup$ – Glen_b Oct 3 '13 at 4:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.