3
$\begingroup$

I have to compare several distributions, but I don't want to use Kolmogorov-Smirnov test, since I'm dealing with big data ($N > 10^{6}$) and traditional hypothesis tests end up with detecting the smallest yet meaningless difference between the two distributions --- i.e. I always reject the null hypothesis, even if a small difference between the two distributions is acceptable considering that I'm analyzing social dynamics/interactions and not physical/biological phenomena.

To overcome such a problem, I was thinking to perform a randomization test where I estimate the intersection area between the densities of the randomized samples. Such a test returns the fraction of times when $(1 - intersection\,\,area)$ is lower than a given epsilon $\epsilon$.

So, for example, given two samples: n <- 1e6 a <- rnorm(n, 0, 1) b <- rnorm(n, 0.2, 1)

with traditional K-S test I reject the null (p < 2.2e-16), whereas with my randomization test with $\epsilon = 0.1$ I can't reject the null (p = 1) and I conclude that the two samples are drawn from the same population (i.e. belong to the same generating process/dynamics) with a given tolerance $\epsilon$. Moreover, such a randomization test does not depend on the samples size, whereas the critical value of K-S test depends on the samples size by a factor of $\sqrt{\frac{n + n'}{nn'}}$.

Do you think there is something inherently wrong in the procedure I have just described?

enter image description here

$\endgroup$
  • 2
    $\begingroup$ Would it be okay for you to estimate a distance between the distributions, instead of a hypothesis test? $\endgroup$ – Dougal May 18 '15 at 5:36
  • 4
    $\begingroup$ If "a small difference is acceptable" you probably don't want an ordinary hypothesis test. Why not compute some measure of the difference? If you really do want a formal test, consider some form of equivalence test. $\endgroup$ – Glen_b May 18 '15 at 5:38
  • $\begingroup$ I would like to do a formal test since I'm writing a research paper and reviewers always ask for p-values/formal tests. I don't know equivalence tests... can you suggest something? Thanks. $\endgroup$ – stochazesthai May 18 '15 at 5:41
  • 3
    $\begingroup$ The logic of this proposal seems to proceed: "When I test for a difference, I find a small one is significant. I don't want small differences to be significant, so I would prefer a test that won't perform well." There are multiple problems with this approach, all related to the attempt to weaken the power of a test in an ad hoc manner. That will force you to work hard to study the power of the proposed test--and your results will likely be even less acceptable to reviewers. I would suggest rethinking this from the outset: why and on what basis are you comparing distributions? $\endgroup$ – whuber May 18 '15 at 14:21
0
$\begingroup$

I suggest to you to use the Cramer-Von Mises test as an alternative to the KS test (it does not take into account the sup of differences and thus will be less sensitive to small "abrupt" differences).

For your big data problem, you could try to use random draws and then determine confidence bands for your test statistic...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.