Two sample Kolmogorov-Smirnov test and p-value interpretation

Question

I have a question regarding the interpretation of resulting p-values of a two sample Kolmogorov Smirnov test.

Basis of my analysis is to try to identify groups that show a difference in their distribution difference compared to totality. I used a two sample Kologorov Smirnov Test in R to do so.

Sample sizes:

Full = 2409
Group_1 = 25
Group_2 = 26
Group_3 = 33
Group_4 = 43

Dataset plots:

"Other" contains a collection of groups containing less than 20 datapoints.

The resulting p-values I get when I compare each "Group" against "Full - Group" are the following:

Group 1: 2.6155e-002
Group 2: 2.1126e-003
Group 3: 7.2113e-002
Group 4: 7.3466e-003

How can I interpret these results - especially with regards to the low number of datapoints per group as well as the difference in sample size for Full (N=2409) and Groups (N=25-43)?

Is the choice of a KS test good or might another test be more appropriate in this case?

Answer 1

If you are using the traditional 0.05 alpha level cutoff then all but group 3 are significantly different from your full group. It is a little easier to see this if the p-values are not in scientific notation ( you can use options(scipen=5) in R to make this less likely). Also group 1 becomes non-significant for some adjustments for multiple tests. You should consider whether that adjustment applies in your case or not. Also note that the groups that are not significant could be different, just low power.

But that just means that any differences, however small, are not easily explained by chance. It could be that your groups are close enough for practical purposes. It is usualy more meaningful to plot the data to see how different the distributions are. You could use the qqplot function as one approach. The vis.test function in the TeachingDemos package for R gives another approach.

One possible hitch is if your groups are part of the "Full" data set as well, then you don't have the independence assumed (but given the sample sizes, I am not sure how much this would affect things). You could address this by taking random samples from the full data set and computing the KS-distance for each (ignore the p-value), then compare where your actual data falls relative to the random samples.

Most of this comes down to what question you really want answered, many of the exact distributional tests answer a different question than the researcher is really interested in.