If I have two samples, with one 30-times larger than the other, which nonparametric test statistics should be considered for checking the null of equal distribution?

I am using Kolmogorov-Smirnov test to check that the two samples come from the same distribution. Any mistake in this? Is there any other test statistics to be used?

  • $\begingroup$ Could you please provide the sample sizes explicitely? Note that you can use other tools to sustain your conclusions. There are other test that you can also consider (see). $\endgroup$
    – user10525
    Nov 6 '12 at 10:17
  • $\begingroup$ @Procrastrinator Sample-A 55, Sample-B 1498. $\endgroup$ Nov 6 '12 at 10:24
  • $\begingroup$ @Procrastinator Yes they are continuous with ties also in the data. For that I am using jitter function and ks.boot function in R to get the most appropriate result. I usually plot boxplot for my analysis. $\endgroup$ Nov 6 '12 at 10:30
  • $\begingroup$ I have used nboots=1000 I will check using 10000 and let you know. $\endgroup$ Nov 6 '12 at 10:52
  • $\begingroup$ @Procratinator No major difference in p-value. earlier-0.12(@ n=1000) Now-0.11(@ n=10000) $\endgroup$ Nov 6 '12 at 10:56

Jittering your data before testing is problematic (jittering is more used for plots so the points are not on top of each other, but the viewer can still understand that there were ties). With jittering you are comparing datasets that are mixtures of the original and the jittering distribution and adding the jittering could artificially increase or decrease the p-value.

The problem with ties in the KS test is that the p-values are computed based on assumptions that no ties are possible, so the work around is to not use the computed p-values but find your own. One possibility is to do a permutation test: run the KS test but ignore the p-value but record the KS test statistic; now combine the 2 sets of data and randomly choose out of the combined data a new set that is the same size as one of the original (and the rest will represent the other group); run the KS test on these new sets and again just record the test statistic. Repeat this process a bunch of times (2,000 or so) and the recorded test statistics will represent the distribution under the null hypothesis, the proportion of the test statistics that are more extreeme than your original one is the p-value.

You mentioned that you thought the null should be rejected based on a plot of the data, perhaps the methods in this paper:

 Buja, A., Cook, D. Hofmann, H., Lawrence, M. Lee, E.-K., Swayne,
 D.F and Wickham, H. (2009) Statistical Inference for exploratory
 data analysis and model diagnostics Phil. Trans. R. Soc. A 2009
 367, 4361-4383 doi: 10.1098/rsta.2009.0120

would appeal to you instead. The simplified version is that they generate several plots of the data under the assumption of the null being true (permuting similar to above) and show those plots along with the same plot of the original data. If you cannot figure out which is the plot of the original data then that supports the null hypothesis, if the plot of the original is very obvious then that argues against the null. The vis.test function in the TeachingDemos package for R helps implement this test.

If you have already decided that there is a difference and just want a p-value less than 0.05 then you can use SnowsPenultimateNormalityTest that is also in the TeachingDemos package (but be warned, the documentation for that function is considered more useful than the function itself).

  • $\begingroup$ Aha, just checked out the code of SnowsPenultimateNormalityTest. Brilliant!! By the way: did that ever convince anyone? I mean, anyone who is just asking for a p-value lower than 0.05? $\endgroup$
    – gui11aume
    Jan 6 '13 at 2:23
  • $\begingroup$ @gui11aume, I think I have convinced a few people to think before doing a normality test, hopefully nobody has actually used my test for a serious study. $\endgroup$
    – Greg Snow
    Jan 7 '13 at 20:50

As fas as I know, there is no particular reason you can't use it with very different sample sizes.

  • $\begingroup$ Thanks for supporting my opinion for K-S test but when I perform this test ,several times I get a p-value of 0.08 to 0.15 which is not considered good for rejecting Null. But by looking the data set we can definitely say that Null should be rejected.That is why I am thinking of using any other test where I can get a p-value less than 0.05. $\endgroup$ Nov 6 '12 at 10:14
  • 2
    $\begingroup$ You said: "But by looking the data set we can definitely say that Null should be rejected" -- assuming you're carrying it out correctly, then you're probably noticing some aspect that the test is less sensitive to than you'd like. That probably suggests a more focused test statistic. ... what causes you to conclude that the null should be rejected? $\endgroup$
    – Glen_b
    Nov 6 '12 at 11:13
  • $\begingroup$ On the biological reasoning and some published reports it should be Null should be rejected. Please have a look at My Data $\endgroup$ Nov 6 '12 at 11:26
  • 2
    $\begingroup$ Those might create and expectation that it would be rejected, but statistical tests look at the data, not biological reasoning or published reports. Rather than get me to download all your data and analyze it (unless you're asking for my consulting services), just draw the two ECDFs... and explain what it is about the data that tells you the null ought to be rejected. $\endgroup$
    – Glen_b
    Nov 6 '12 at 11:37
  • $\begingroup$ @aravindramesh ... that way, we can try to tease out what it is you regard as a relevant difference, which would allow an improvement in power against the alternatives you see as important. $\endgroup$
    – Glen_b
    Nov 6 '12 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.