Reducing sample size for meaningful p-values

I'm analyzing correlations (using phi and chi-square) among pairs of items within a large dataset (1.2 million records).

My understanding is that it's more likely to obtain spurious associations that are statistically significant when working with big data. With that in mind, I'm wondering if it's recommended to take several random samples of a much smaller size for performing my correlation analysis? I could then summarize the distribution of estimates I obtain throughout the random samples using the mean and confidence intervals or the median and IQR.

If this is recommended are there any good references for such a technique? If I use such a resampling technique, how should I calculate the sample size so that the p-values I obtain are meaningful?

• If you regard it as a problem that hypothesis tests are consistent (that with a large enough sample they will reject even a slightly false null), that's a strong sign that such hypothesis tests are completely the wrong tool from your problem at every sample size. It's important to clearly define what you're trying to find out before you start with the statistics. Feb 4, 2020 at 12:23

Remember that a p-value does not tell you how different the truth is from what the null hypothesis asserts. What you’re looking for is called effect size.

You’re allowed to get a statistically significant result and say, “Okay, but it isn’t interesting enough for me to care,” and this does not make you a lazy scientist or negligent statistician.

So you’re allowed to say that your gigantic data set gave a very small p-value, one that convinces you that the null hypothesis is wrong, but that the effect is so small that you don’t care.

• So are you recommending that I set a certain cut-off for effect size? I understand the difference between p-values and effect sizes but what I'm looking for is a way to adjust for the increased power due to the large dataset Feb 4, 2020 at 1:39
• The problem that you have is that the sample size allows you to notice differences that are subtle, perhaps much too subtle to be interesting. Therefore, yes, only care about an effect size large enough to be interesting.
– Dave
Feb 4, 2020 at 1:47
• Do you have a reference that I could cite for using effect size cut-offs, rather than statistical significance cut-offs? Feb 4, 2020 at 16:15
• @bambi Fritz, Catherine O., Peter E. Morris, and Jennifer J. Richler. "Effect size estimates: current use, calculations, and interpretation." Journal of experimental psychology: General 141.1 (2012): 2.
– Dave
Feb 4, 2020 at 16:22

The phi you are calculating is an effect size statistic. If you were to take a perfect subsample of your whole sample, the phi statistic would come out the same. So there is no benefit to doing such a sub-sampling when considering this statistic. That being said, it make be helpful to look at the confidence interval for phi.

I don't think using the sub-sampling technique would make the p values any more "meaningful". They are meaningful for the whole sample. But they mean what they mean, and not something else.