When working on very large datasets, identifying effects rests more on quantification than significance and many questions/answers give insight on what to do regarding large samples like this (very good) answer.
Although deciding if an effect is quantitatively interesting can be rather easily done for some tests (e.g mean difference and such), I can't say the same for others, so I'm trying to go for a sampling approach, but have little experience as to what this implies for analysis.
I'm trying to compare the (near identical) distributions of 2 samples, both of which have large sample sizes (~20 000 each). I'm pretty sure that the 2 samples are from an equal distribution, but using a Kolmogorov-Smirnov test will almost systematically return near-zero p-values with this sample size, so I want to sub-sample each of these 2 initial samples and compare the subs (which I make ~1000 values). Doing this a few times gave me the expected results -- no significant difference between the 2 -- most of the time (~92%).
How do I go about rigorously doing and explaining this? Saying "The test was done on subsamples n times and came out as negative x% of the time" seems wrong. Also, when cross-validating/bootstrapping something, there are bound to be repeated tests so do I need to do p-value adjustments on these?
