I'm trying to compare two samples to see if they come from same distribution. I'm not just interested in the measures of central tendency but in the overall shape and nature of the distributions. Based on what I have read on CV and elsewhere, K-S test seems like a good fit for the data I have.
The test, however, is very sensitive to the differences between the two. There are many differences it detects with significant p-values (< 1e-3) that are not really practically significant.(1) The ones I'm really looking for have much lower p-values (< 1e-10), but I'm not sure how to justify using an arbitrarily arrived threshold.
Are there ways to find out a threshold or create a transform function on p-values to have a reasonable cutoff? Any other methods I can use in tandem? Compute an empirical p-value somehow?
Note that the two samples have ties.
As it seems from the initial answers, this is problem that requires understanding the data itself so as to be able to distinguish between practically significant versus statistically significant differences. It'd be great if one can explain general strategies to overcome such situation.