Is there a multiple-sample version or alternative to the Kolmogorov-Smirnov Test?

Question

I am comparing the size distribution of trees in six pairs of plots where one plot received a treatment and the other a control. Using a Kolmogorov-Smirnov test on each pair of plots I find that $p$ ranges from $0.0003707$ to $0.75$. Are there any appropriate methods for dealing with all of the replicates together, such as a multi-sample extension of the KS test, or is there an appropriate follow up test? Or should I just conclude something like "The size distribution differ significantly $(p < 0.05$) in 2 pairs of plots and marginally ($p = 0.59$) in one pair of plots."

Answer 1

There actually are some multiple sample KS Tests. E.g., an r-sample Kolmogorov-Smirnov-Test with $r\geq 2$ which, I believe, has good power. A preprint of that beautiful paper is available here. I also know of K-Sample Analogues of the Kolmogorov-Smirnov and Cramer-V. Mises Tests (but they have less power as far as I know).

Answer 2

There is an R package kSamples that gives you, among other things, a non-parametric k-sample Anderson-Darling test. The null hypothesis is that all k samples came from the same distribution which does not need to be specified. Maybe you can use this.

Little example on comparing Normal and Gamma-distributed samples scaled so that they have the same mean and variance:

library("kSamples")
set.seed(142)
samp.num <- 100
alpha <- 2.0; theta <- 3.0  # Gamma parameters shape and scale, using Wikipedia notation
gam.mean <- alpha * theta # mean of the Gamma
gam.sd <- sqrt(alpha) * theta # S.D. of the Gamma
norm.data <- rnorm(samp.num, mean=gam.mean, sd=gam.sd)  # Normal with the same mean and SD as the Gamma
gamma.data <- rgamma(samp.num, shape=alpha, scale=theta)
norm.data2 <- rnorm(samp.num, mean=gam.mean, sd=gam.sd)
norm.data3 <- rnorm(samp.num, mean=gam.mean, sd=gam.sd)
ad.same <- ad.test(norm.data,norm.data2,norm.data3) # "not significant, p ~ 0.459"
ad.diff <- ad.test(gamma.data,norm.data2,norm.data3) # "significant, p ~ 0.00066"

Answer 3

A couple of approaches:

Use the pairwise p-values but adjust them for multiple comparisons using something like the Bon Feroni or False Discovery Rate adjustmetns (the first will probably be a bit over conservative). Then you can be confident that any that are still significantly different are probably not due to the multiple testing.

You could create an overall test in the flavor of KS by finding the greatest distance between any of the distributions, i.e. plot all the empirical cdf's and find the largest distance from the bottommost line to the topmost line, or maybe average distance or some other meaningful measure. Then you can find if that is significant by doing a permutation test: group all the data into 1 big bin, then randomly split it into groups with the same sample sizes as your original groups, recompute the stat on the permuted data and repeat the process many times (999 or so). Then see how your original data compares to the permuted data sets. If the original data statistic fall in the middle of the permuted ones then there is no significant differences found, but if it is at the edge, or beyond any of the permuted ones then there is something significant going on (but this does not tell you which are different). You should probably try this out with simulated data where you know there is a difference that is big enough to be interesting just to check the power of this test to find the interesting differences.