# Does rejecting a study's credibility based on sample size make sense even if statistical significance is reached?

I've seen people do this commonly, and to me, it makes no sense because one of the main reasons of having a large enough sample size is that you CAN reach statistical significance in the first place.

• A key assumption, difficult to satisfy in practice, is that the sample be randomly sampled. With suitable protocols in place, it is easier to believe that a large sample achieved randomness than that a small one did. (This may not be true, but perception is important.)// Also, for evidence that a drug is safe and effective, a large sample enables inclusion of a diverse group so that problematic unintended effects in subpopulations are more likely to be discovered. Jun 24 at 5:14
• Your point has merit, but as a (simplistic) example for clinical trials, 100 successes in 100 might give 95% Bayesian credible interval (.964, .9999) , while 995 in 1000 might give interval (.988, .9978). Even though I believe the math, I wouldn't fault someone who feels much more comfortable with the larger trial. Jun 24 at 5:14
• @BruceET Okay, but perception is hardly reality. That's kind of what I'm railing against in my post, is an incorrect perception. Jun 24 at 13:40
• @BruceET I'm not faulting them. I'm asking if it makes sense, statistically. And I don't see that you're making an argument for it, other than appealing to how they feel about it. Jun 24 at 13:54
• I guess it comes down to what you mean by "makes sense statistically." Jun 24 at 14:25

This is to some extent answered in this paper. What it describes is a setting, where

• We (sort of randomly) pick research hypothesis to investigate, for which the null hypothesis is either exactly true or we have the hypothesized effect size.
• Of course, we don't know for sure when we start our study.
• In that setting, a "small sample size means smaller power and [...] the PPV [positive preditive value] for a true research finding" when observing $$p\leq 0.05$$ decreases as power decreases.
• I.e. how likely it is that $$p\leq 0.05$$ (or any other threshold $$\alpha$$) indicates a correct rejection of the null hypothesis depends on the power. The other thing that matters is what proportion of investigated alternative hypotheses are true (or in practice, how implausible your research finding is a-priori).
• This means surprising findings with $$p\leq \alpha$$ from small studies are much less likely to be true than expected findings from very large studies.
• The example in the essay is overly simplified, because few point null hypotheses are exactly true and even the meaningfully large effects follow some kind of distribution, but the underlying point is true.

Why does this happen? Well, the p-value is random variable that is in turn based on the other random variables in your study. The smaller the sample size, the noisier the p-value is. Consider an extreme case:

• Let's assume we have absolutely no data.
• A valid null hypothesis test in that setting is to reject the null hypothesis randomly with probability $$\alpha$$.
• If we know nothing else, then the probability that a study with $$p\leq \alpha$$ has produced a true research finding is then simply the a-priori probability that the alternative hypothesis was true.

You may think I'm being silly here to do a null hypothesis test without any data, but when I do a study with a power that is really low (=I have hardly any data), then I am essentially doing the same thing (i.e. looking at a random variable that has very little connection with my research question). That's what the paper I linked pointed out.

• Your argument of power doesn't make sense to me. Power is the probability of correctly rejecting the null hypothesis; we agree there. But if you've already rejected the null hypothesis, what role does the 'probability of rejecting' it have when it's already occurred. Power is useful for deciding how many samples you want in your study in advance so that you can be reasonably sure you'll reject the null if the alternative hypothesis is true. What the role of power is not is to dismiss an already statistically significant result retroactively. Jun 24 at 13:52
• It's kind of like watching someone make a three-pointer in basketball and then retroactively claiming they didn't because the probability of making the shot was low. How does that make sense? Jun 24 at 13:53
• No, we don't agree on the definition of power: It's the probability of rejecting the null hypothesis conditional on it being wrong. Similarly, the p-value is not the probability that the null hypothesis is true, but the probability to reject the null hypothesis, if it is true. To stick to basketball: Do you accept I'm the world's best 3-point shooter? I'll send you 50 videos of successes & you'll have no idea how many videos I took to get 50. Does it matter for this that I don't have a NBA contract & last played recreationally in college? And, I believe what I describe is widely accepted. Jun 24 at 14:03
• That is what I mean by power. We don't disagree, you've even taken that shortcut in defining it in your own post: "how likely it is that p≤0.05 (or any other threshold α) indicates a correct rejection of the null hypothesis depends on the power." I only have a limited amount of characters to work with. We don't disagree on the definitions of power or p-value. I've alluded to the full definition of power in this quote "Power is useful for deciding how many samples you want in your study in advance so that you can be reasonably sure you'll reject the null if the alternative hypothesis is true." Jun 24 at 14:06
• No, I did not define it that way. Jun 24 at 14:09