# Two sample Kolmogorov-Smirnov test and p-value interpretation

I have a question regarding the interpretation of resulting p-values of a two sample Kolmogorov Smirnov test.

Basis of my analysis is to try to identify groups that show a difference in their distribution difference compared to totality. I used a two sample Kologorov Smirnov Test in R to do so.

Sample sizes:

Full = 2409
Group_1 = 25
Group_2 = 26
Group_3 = 33
Group_4 = 43


Dataset plots:

"Other" contains a collection of groups containing less than 20 datapoints.

The resulting p-values I get when I compare each "Group" against "Full - Group" are the following:

Group 1: 2.6155e-002
Group 2: 2.1126e-003
Group 3: 7.2113e-002
Group 4: 7.3466e-003


How can I interpret these results - especially with regards to the low number of datapoints per group as well as the difference in sample size for Full (N=2409) and Groups (N=25-43)?

Is the choice of a KS test good or might another test be more appropriate in this case?

## 1 Answer

If you are using the traditional 0.05 alpha level cutoff then all but group 3 are significantly different from your full group. It is a little easier to see this if the p-values are not in scientific notation ( you can use options(scipen=5) in R to make this less likely). Also group 1 becomes non-significant for some adjustments for multiple tests. You should consider whether that adjustment applies in your case or not. Also note that the groups that are not significant could be different, just low power.

But that just means that any differences, however small, are not easily explained by chance. It could be that your groups are close enough for practical purposes. It is usualy more meaningful to plot the data to see how different the distributions are. You could use the qqplot function as one approach. The vis.test function in the TeachingDemos package for R gives another approach.

One possible hitch is if your groups are part of the "Full" data set as well, then you don't have the independence assumed (but given the sample sizes, I am not sure how much this would affect things). You could address this by taking random samples from the full data set and computing the KS-distance for each (ignore the p-value), then compare where your actual data falls relative to the random samples.

Most of this comes down to what question you really want answered, many of the exact distributional tests answer a different question than the researcher is really interested in.

• Thank you very much for your answer! I updated the entry with a plot of the distributions (wasn't allowed to do so in the first place). The reason for my question was actually the visualisation of the distributions: Although using the 0.05 cutoff three distributions would be considered significantly different, the boxplots still show some considerable overlap between them. I removed the datapoints from each "Group" from "Full" before doing the calculation. The random sampling approach sounds like a very interesting idea! – Matthias May 13 '11 at 14:07
• It looks like this comes down to the question of practical vs statistical significance. Visually you are not seeing much difference between the groups, but the tests are significant. You need to ask yourself if they are close enough for what you want to do and worry less about tests of very narrow hypotheses. Consider the 2 distributions, one is uniform from 0 to 0.99, the other is uniform from 0.01 to 1, are they different, the tests can say yes given enough data, but for most practical considerations the differences won't matter. – Greg Snow May 13 '11 at 16:11