How to correct for small p-value due to very large sample size

I am running into a problem where an independent variable, which should have no predictive power on the dependent variable based on domain knowledge, comes out with very small p-value because the sample size is very large(~100,000). If I only use < 5000 data points, then the p-value becomes large enough to support the prior that the variable is insignificant. However, I don't think tweaking the sample size to get to the desired conclusion is a good practice. Is there any procedure to adjust for small p-value simply due to huge sample size?

• When the p-value is small, chance is not a good explanation for the estimated relationship. That's it, period: no adjustment is needed. Perhaps you should be focusing on effect size or measures of importance in your application rather than p-value. See this related question, too.
– whuber
Nov 26, 2012 at 22:53
• The problem is I am working with economic models, where a R^2 of 0.01 is considered "very" high. Is there an effect size analysis that would be appropriate for such low signal-to-noise ratio problems?
– user13587
Nov 26, 2012 at 23:47
• It's the one you do when reviewing literature and the effects you found. Maybe this effect is in the range of meaningful sizes, maybe not. You have to assess that. Maybe your prediction of no effect was based on the fact that it was just really small. If it's actually a meaningful amount then you've got an explanation for why others didn't find it.
– John
Nov 27, 2012 at 0:10

You're valuing the p-value far too highly. Report the magnitude of the statistically significant effect. It should be such a small value that statistically significant is pretty irrelevant. In terms of something like Cohen's d, it takes an effect size of about 0.006 to need N's that large to be found. Talk about the effect size reasonably. That's what you should be doing for all of your effects, significant or not, expected or not.

• Cohen's D works well for tests that look at means, but if you're running nonparametric tests, say Wilcoxon rank sum, there is no (equivalent) effect size measure. How would you then treat that situation?
– Jon
Dec 8, 2016 at 23:12
• I crafted a response but as it got more involved I decided the better thing to do is suggest you direct this to SE after a careful look at the current questions on effect size and nonparametric tests. I only will say here that Cohen's D is only mentioned for convenience sake as the questioner gives no information on what the actual measures are.
– John
Dec 9, 2016 at 3:13
• Well, for general nonparametric methods, there seems to be a lack of effect size measures. I've resulted to using Cohen's D or a bootstrapped Cohen's D. There have been other threads that have attempted at this as well stats.stackexchange.com/questions/133077/…
– Jon
Dec 9, 2016 at 17:10

A few years later... The paper by Naaman (2016) Almost sure hypothesis testing and a resolution of the Jeffreys-Lindley paradox seems relevant. From the abstract:

A new method of hypothesis testing is proposed ensuring that as the sample size grows, the probability of a type I error will become arbitrarily small by allowing the significance level to decrease with the number of observations in the study.

• The "it is likely" statement is incorrect in general. If, for example, one was working in a field in which large numbers of false positive signals were generated, chance might be at least as good an explanation for a small p-value as a real relationship. We cannot trust $p$-values to be the sole arbiters of truth. Nov 26, 2012 at 23:32