7
$\begingroup$

I am running into a problem where an independent variable, which should have no predictive power on the dependent variable based on domain knowledge, comes out with very small p-value because the sample size is very large(~100,000). If I only use < 5000 data points, then the p-value becomes large enough to support the prior that the variable is insignificant. However, I don't think tweaking the sample size to get to the desired conclusion is a good practice. Is there any procedure to adjust for small p-value simply due to huge sample size?

$\endgroup$
  • 6
    $\begingroup$ When the p-value is small, chance is not a good explanation for the estimated relationship. That's it, period: no adjustment is needed. Perhaps you should be focusing on effect size or measures of importance in your application rather than p-value. See this related question, too. $\endgroup$ – whuber Nov 26 '12 at 22:53
  • $\begingroup$ The problem is I am working with economic models, where a R^2 of 0.01 is considered "very" high. Is there an effect size analysis that would be appropriate for such low signal-to-noise ratio problems? $\endgroup$ – user13587 Nov 26 '12 at 23:47
  • $\begingroup$ It's the one you do when reviewing literature and the effects you found. Maybe this effect is in the range of meaningful sizes, maybe not. You have to assess that. Maybe your prediction of no effect was based on the fact that it was just really small. If it's actually a meaningful amount then you've got an explanation for why others didn't find it. $\endgroup$ – John Nov 27 '12 at 0:10
16
$\begingroup$

You're valuing the p-value far too highly. Report the magnitude of the statistically significant effect. It should be such a small value that statistically significant is pretty irrelevant. In terms of something like Cohen's d, it takes an effect size of about 0.006 to need N's that large to be found. Talk about the effect size reasonably. That's what you should be doing for all of your effects, significant or not, expected or not.

$\endgroup$
  • $\begingroup$ Cohen's D works well for tests that look at means, but if you're running nonparametric tests, say Wilcoxon rank sum, there is no (equivalent) effect size measure. How would you then treat that situation? $\endgroup$ – Jon Dec 8 '16 at 23:12
  • $\begingroup$ I crafted a response but as it got more involved I decided the better thing to do is suggest you direct this to SE after a careful look at the current questions on effect size and nonparametric tests. I only will say here that Cohen's D is only mentioned for convenience sake as the questioner gives no information on what the actual measures are. $\endgroup$ – John Dec 9 '16 at 3:13
  • $\begingroup$ Well, for general nonparametric methods, there seems to be a lack of effect size measures. I've resulted to using Cohen's D or a bootstrapped Cohen's D. There have been other threads that have attempted at this as well stats.stackexchange.com/questions/133077/… $\endgroup$ – Jon Dec 9 '16 at 17:10
  • $\begingroup$ I put a little answer in there that might help you. $\endgroup$ – John Dec 10 '16 at 16:57
0
$\begingroup$

Keep in mind the truth that "correlation is not causation". That is, while you might not see WHY there should be a predictive relationship, there nonetheless is a strong enough correlation that you receive a small p-value.

As an example: I could perform a health & fitness study on kids who weigh between 75 - 100 lbs. If I also write down their ages & their heights, I would find correlations between both age & height to health as well as weight to health. Why? Well, if I grab a 4 year old who weighs 75 pounds, s/he is likely unhealthy. Similarly if I grab a boy 6'2" who is 100 lbs. This makes sense to you because it is intuitive. The effects in the case you're studying are likely less intuitive.
Why did this occur? Because I put a constraint on one correlated variable which inherently creates relationships with the others. (If I took all weights, then the 6'2" 100 lb boy would be drown out by other 6'2" kids who weigh much more.)

Lastly, keep in mind that the p-value you are calculating is against the null hypothesis (or so I presume). The null hypothesis is merely testing against absolutely no effect. So if you have a mild effect that is so trivial you think is uninteresting, it will still have a low p-value. So as others said, p-value alone is not the end-all-be-all. It is merely a good filter & starting point.

$\endgroup$
-2
$\begingroup$

If the p-value indicates a relationship, then it is likely that the relationship does exist. However, I wholeheartedly agree with the previous response, that you should be focussing on a measure of the effect of the relationship.

$\endgroup$
  • 8
    $\begingroup$ The "it is likely" statement is incorrect in general. If, for example, one was working in a field in which large numbers of false positive signals were generated, chance might be at least as good an explanation for a small p-value as a real relationship. We cannot trust $p$-values to be the sole arbiters of truth. $\endgroup$ – guest Nov 26 '12 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy