When working on very large datasets, identifying effects rests more on quantification than significance and many questions/answers give insight on what to do regarding large samples like this (very good) answer.

Although deciding if an effect is quantitatively interesting can be rather easily done for some tests (e.g mean difference and such), I can't say the same for others, so I'm trying to go for a sampling approach, but have little experience as to what this implies for analysis.

I'm trying to compare the (near identical) distributions of 2 samples, both of which have large sample sizes (~20 000 each). I'm pretty sure that the 2 samples are from an equal distribution, but using a Kolmogorov-Smirnov test will almost systematically return near-zero p-values with this sample size, so I want to sub-sample each of these 2 initial samples and compare the subs (which I make ~1000 values). Doing this a few times gave me the expected results -- no significant difference between the 2 -- most of the time (~92%).

How do I go about rigorously doing and explaining this? Saying "The test was done on subsamples n times and came out as negative x% of the time" seems wrong. Also, when cross-validating/bootstrapping something, there are bound to be repeated tests so do I need to do p-value adjustments on these?

  • 1
    $\begingroup$ This approach seems strange to me. Maybe estimate the treatment effect, and present a confidence interval? Then ask yourself if any of the values within that CI have any practical significance in your application area? $\endgroup$ – kjetil b halvorsen Jul 1 '18 at 11:30

I am not really sure whether the methods I am exploring in this answer is common or not. But I will give it a try and share what I think.

I believe your problem can be broken into two parts:

a) Explaining things using only the p-values:

Boxplot of the p-values can be shown to show the distribution of p-values. You can also plot the frequency distribution, etc.

b) Reporting something over how many times the p-value was significant.

You can measure how many times the p-value was significant, for ex- less than 0.05. This random variable will follow a binomial distribution. You can estimate the parameters of this distribution and also compute some descriptive statistics from the sample over which this distribution has been estimated.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.