Hypothesis testing on subsamples and cross validation

Question

When working on very large datasets, identifying effects rests more on quantification than significance and many questions/answers give insight on what to do regarding large samples like this (very good) answer.

Although deciding if an effect is quantitatively interesting can be rather easily done for some tests (e.g mean difference and such), I can't say the same for others, so I'm trying to go for a sampling approach, but have little experience as to what this implies for analysis.

I'm trying to compare the (near identical) distributions of 2 samples, both of which have large sample sizes (~20 000 each). I'm pretty sure that the 2 samples are from an equal distribution, but using a Kolmogorov-Smirnov test will almost systematically return near-zero p-values with this sample size, so I want to sub-sample each of these 2 initial samples and compare the subs (which I make ~1000 values). Doing this a few times gave me the expected results -- no significant difference between the 2 -- most of the time (~92%).

How do I go about rigorously doing and explaining this? Saying "The test was done on subsamples n times and came out as negative x% of the time" seems wrong. Also, when cross-validating/bootstrapping something, there are bound to be repeated tests so do I need to do p-value adjustments on these?

Answer 1

Probably just stating the obvious here, but if the two samples are actual from the same distribution, then it shouldn't matter how large your sample size is. Your p-values will mostly come back larger than 0.05.

data1 <- rnorm(20000, 0, 5)
data2 <- rnorm(20000, 0, 5)

ks.test(data1, data2)

Even a difference in means of say 0.1 isn't usually going to be picked up with 20,000 samples. This returns mostly p-values larger than 0.05.

data1 <- rnorm(20000, 0, 5)
data2 <- rnorm(20000, 0.1, 5)

ks.test(data1, data2)

On the other hand, if you take small enough samples from your samples, even if your two samples are from distinctly different distributions, your p-values will come back mostly positive.

data1 <- rnorm(40, 0, 5)
data2 <- rnorm(40, 0, 4)

ks.test(data1, data2)

Maybe your problem is better suited for some approach other than null hypothesis testing and p-values if you just want to establish that the distributions are similar enough for your purposes but not identical.

Answer 2

I am not really sure whether the methods I am exploring in this answer is common or not. But I will give it a try and share what I think.

I believe your problem can be broken into two parts:

a) Explaining things using only the p-values:

Boxplot of the p-values can be shown to show the distribution of p-values. You can also plot the frequency distribution, etc.

b) Reporting something over how many times the p-value was significant.

You can measure how many times the p-value was significant, for ex- less than 0.05. This random variable will follow a binomial distribution. You can estimate the parameters of this distribution and also compute some descriptive statistics from the sample over which this distribution has been estimated.