Skip to main content
1 of 2
Cliff AB
  • 21.6k
  • 1
  • 51
  • 104

What we call P-hacking is applying a significance test multiple times and only reporting the significance results. Whether this is good or bad is situationally dependent.

To explain, let's think about true effects in Bayesian terms, rather than null and alternative hypotheses. As long as we believe our effects of interest come from a continuous distribution, then we know the null hypothesis is false. However, in the case of a two-sided test, we don't know whether it is positive or negative. Under this light, we can think of p-values for two sided tests as a measure of how strong the evidence is that our estimate has the correct direction (i.e., positive or negative effect).

Under this interpretation, any significance test can have three possible outcomes: we see enough evidence to conclude the direction of the effect and we are correct, we see enough evidence to conclude the direction of the effect but we are wrong, or we don't see enough evidence to conclude the direction of the effect. Note that conditional that you have enough evidence (i.e., $p < \alpha$), the probability of getting the direction correct should be greater than the probability of getting it incorrect (unless you have some really crazy, really bad test), although as the true effect size approaches zero, the conditional probability of getting the direction correct given sufficient evidence approaches 0.5.

Now, consider what happens when you keep going back to get more data. Each time you get more data, your probability of getting the direction correct conditional on sufficient data only goes up. So under in this scenario, we should realize that by getting more data, although we are in fact increasing the probability of a type I error, we are also reducing the probability of mistakenly concluding the wrong direction.

Take this in contrast the more typical abuse of P-hacking; we test 100's of effect sizes that have good probability of being very small and only report the significant ones. Note that in this case, if all the effects are small, we have a near 50% chance of getting the direction wrong when we declare significance.

And finally, consider the implications of not getting more data when we have an insignificant result. That would imply never collecting more information on the topic, which won't really push the science forward, would it? One underpowered study would kill a whole field.

Cliff AB
  • 21.6k
  • 1
  • 51
  • 104