Reducing sample size for meaningful p-values I'm analyzing correlations (using phi and chi-square) among pairs of items within a large dataset (1.2 million records).
My understanding is that it's more likely to obtain spurious associations that are statistically significant when working with big data. With that in mind, I'm wondering if it's recommended to take several random samples of a much smaller size for performing my correlation analysis? I could then summarize the distribution of estimates I obtain throughout the random samples using the mean and confidence intervals or the median and IQR.
If this is recommended are there any good references for such a technique? If I use such a resampling technique, how should I calculate the sample size so that the p-values I obtain are meaningful?
 A: Remember that a p-value does not tell you how different the truth is from what the null hypothesis asserts. What you’re looking for is called effect size.
You’re allowed to get a statistically significant result and say, “Okay, but it isn’t interesting enough for me to care,” and this does not make you a lazy scientist or negligent statistician.
So you’re allowed to say that your gigantic data set gave a very small p-value, one that convinces you that the null hypothesis is wrong, but that the effect is so small that you don’t care.
A: The phi you are calculating is an effect size statistic.  If you were to take a perfect subsample of your whole sample, the phi statistic would come out the same.  So there is no benefit to doing such a sub-sampling when considering this statistic.  That being said, it make be helpful to look at the confidence interval for phi.
I don't think using the sub-sampling technique would make the p values any more "meaningful". They are meaningful for the whole sample. But they mean what they mean, and not something else.
