# K-S test is too sensitive

I'm trying to compare two samples to see if they come from same distribution. I'm not just interested in the measures of central tendency but in the overall shape and nature of the distributions. Based on what I have read on CV and elsewhere, K-S test seems like a good fit for the data I have.

The test, however, is very sensitive to the differences between the two. There are many differences it detects with significant p-values (< 1e-3) that are not really practically significant.(1) The ones I'm really looking for have much lower p-values (< 1e-10), but I'm not sure how to justify using an arbitrarily arrived threshold.

Are there ways to find out a threshold or create a transform function on p-values to have a reasonable cutoff? Any other methods I can use in tandem? Compute an empirical p-value somehow?

Note that the two samples have ties.

Thanks!

Edit:

As it seems from the initial answers, this is problem that requires understanding the data itself so as to be able to distinguish between practically significant versus statistically significant differences. It'd be great if one can explain general strategies to overcome such situation.

• Out of curiosity, how many observations do you have for each sample? – Jonathan Lisic Jun 28 '16 at 20:39
• @JonathanLisic About 500. – Vivek Rai Jun 28 '16 at 21:34
• If you use a wrench to hammer in screws, don't complain that the resulting mess is because something is wrong with the wrench or the screws ... if you care about "practical significance" don't do an ordinary hypothesis test, because it simply doesn't deal with that issue. It sounds like you're interested in something like an effect size, so specify what you want to identify and measure that. – Glen_b -Reinstate Monica Jun 29 '16 at 0:05
• @Glen_b Yes, you are correct. I've only started to think about the problem and that's what I am trying to understand -- what I want to identity and measure that. It is in the same process that K-S test came up. Can you talk a bit about general strategies to do what you suggest? Thanks! – Vivek Rai Jun 29 '16 at 6:39
• You're asking me for a general strategy for figuring out what you want? My general strategy is to ask something like "what are you really trying to achieve?" – Glen_b -Reinstate Monica Jun 29 '16 at 6:44

You have run into one of the many problems with null-hypothesis significance testing. Specifically, the mere falsity of the null hypothesis doesn't say much about a population. A test has told you that two samples aren't from literally identical populations, but that's not what you actually care about. What you actually care about is up to you; it sounds like you care about how different the populations are. The solution is not to try to transform $p$-values ("when all you have is a hammer", etc.) but to use a more appropriate technique. One tack you could take is to use each sample to estimate a population distribution, as with kernel density estimation, then compare your estimates to each other. But it really depends on your specific data and the specific problem you want to solve.