Test to compare distributions using the intersection area of the densities

I have to compare several distributions, but I don't want to use Kolmogorov-Smirnov test, since I'm dealing with big data ($N > 10^{6}$) and traditional hypothesis tests end up with detecting the smallest yet meaningless difference between the two distributions --- i.e. I always reject the null hypothesis, even if a small difference between the two distributions is acceptable considering that I'm analyzing social dynamics/interactions and not physical/biological phenomena.

To overcome such a problem, I was thinking to perform a randomization test where I estimate the intersection area between the densities of the randomized samples. Such a test returns the fraction of times when $(1 - intersection\,\,area)$ is lower than a given epsilon $\epsilon$.

So, for example, given two samples: n <- 1e6 a <- rnorm(n, 0, 1) b <- rnorm(n, 0.2, 1)

with traditional K-S test I reject the null (p < 2.2e-16), whereas with my randomization test with $\epsilon = 0.1$ I can't reject the null (p = 1) and I conclude that the two samples are drawn from the same population (i.e. belong to the same generating process/dynamics) with a given tolerance $\epsilon$. Moreover, such a randomization test does not depend on the samples size, whereas the critical value of K-S test depends on the samples size by a factor of $\sqrt{\frac{n + n'}{nn'}}$.

Do you think there is something inherently wrong in the procedure I have just described? • Would it be okay for you to estimate a distance between the distributions, instead of a hypothesis test? – Dougal May 18 '15 at 5:36
• If "a small difference is acceptable" you probably don't want an ordinary hypothesis test. Why not compute some measure of the difference? If you really do want a formal test, consider some form of equivalence test. – Glen_b May 18 '15 at 5:38
• I would like to do a formal test since I'm writing a research paper and reviewers always ask for p-values/formal tests. I don't know equivalence tests... can you suggest something? Thanks. – stochazesthai May 18 '15 at 5:41
• The logic of this proposal seems to proceed: "When I test for a difference, I find a small one is significant. I don't want small differences to be significant, so I would prefer a test that won't perform well." There are multiple problems with this approach, all related to the attempt to weaken the power of a test in an ad hoc manner. That will force you to work hard to study the power of the proposed test--and your results will likely be even less acceptable to reviewers. I would suggest rethinking this from the outset: why and on what basis are you comparing distributions? – whuber May 18 '15 at 14:21