I am comparing the size distribution of trees in six pairs of plots where one plot received a treatment and the other a control. Using a Kolmogorov-Smirnov test on each pair of plots I find that $p$ ranges from $0.0003707$ to $0.75$. Are there any appropriate methods for dealing with all of the replicates together, such as a multi-sample extension of the KS test, or is there an appropriate follow up test? Or should I just conclude something like "The size distribution differ significantly $(p < 0.05$) in 2 pairs of plots and marginally ($p = 0.59$) in one pair of plots."

  • 3
    $\begingroup$ What is it you want to compare about these distributions, that they differ in central tendency, or that they differ in shape? I tend to think of KS as being more about the shape / nature of a distribution, but something like the Friedman test can determine that the samples differ in central tendency. $\endgroup$ Aug 31 '12 at 19:07

There actually are some multiple sample KS Tests. E.g., an r-sample Kolmogorov-Smirnov-Test with $r\geq 2$ which, I believe, has good power. A preprint of that beautiful paper is available here. I also know of K-Sample Analogues of the Kolmogorov-Smirnov and Cramer-V. Mises Tests (but they have less power as far as I know).

  • 5
    $\begingroup$ Well, the disadvantage of that "beautiful" paper by Böhm and Hornik is that there is no publicly available implementation available as far as I can tell. The maths is complex enough that you wouldn't like to implement it yourself. I mailed the authors and asked them but they did not reply. Note that Hornik is a member of the R Core Developers group... If someone knows about an implementation, pls post a link here! $\endgroup$ Aug 11 '15 at 14:12

There is an R package kSamples that gives you, among other things, a non-parametric k-sample Anderson-Darling test. The null hypothesis is that all k samples came from the same distribution which does not need to be specified. Maybe you can use this.

Little example on comparing Normal and Gamma-distributed samples scaled so that they have the same mean and variance:

samp.num <- 100
alpha <- 2.0; theta <- 3.0  # Gamma parameters shape and scale, using Wikipedia notation
gam.mean <- alpha * theta # mean of the Gamma
gam.sd <- sqrt(alpha) * theta # S.D. of the Gamma
norm.data <- rnorm(samp.num, mean=gam.mean, sd=gam.sd)  # Normal with the same mean and SD as the Gamma
gamma.data <- rgamma(samp.num, shape=alpha, scale=theta)
norm.data2 <- rnorm(samp.num, mean=gam.mean, sd=gam.sd)
norm.data3 <- rnorm(samp.num, mean=gam.mean, sd=gam.sd)
ad.same <- ad.test(norm.data,norm.data2,norm.data3) # "not significant, p ~ 0.459"
ad.diff <- ad.test(gamma.data,norm.data2,norm.data3) # "significant, p ~ 0.00066"

A couple of approaches:

Use the pairwise p-values but adjust them for multiple comparisons using something like the Bon Feroni or False Discovery Rate adjustmetns (the first will probably be a bit over conservative). Then you can be confident that any that are still significantly different are probably not due to the multiple testing.

You could create an overall test in the flavor of KS by finding the greatest distance between any of the distributions, i.e. plot all the empirical cdf's and find the largest distance from the bottommost line to the topmost line, or maybe average distance or some other meaningful measure. Then you can find if that is significant by doing a permutation test: group all the data into 1 big bin, then randomly split it into groups with the same sample sizes as your original groups, recompute the stat on the permuted data and repeat the process many times (999 or so). Then see how your original data compares to the permuted data sets. If the original data statistic fall in the middle of the permuted ones then there is no significant differences found, but if it is at the edge, or beyond any of the permuted ones then there is something significant going on (but this does not tell you which are different). You should probably try this out with simulated data where you know there is a difference that is big enough to be interesting just to check the power of this test to find the interesting differences.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.