# Studies with small sample sizes

I'm asking myself the question of why studies with small sample sizes are not as convincing as those with larger sample sizes, and when this becomes a statistical issue. A complaint I've heard a lot with studies is while they show a certain "desired" result (e.g. the treatment group had lower cholesterol than the placebo group), the sample size was small.

Statistically, if you have the sample size to detect an effect, then the sample size was large enough in that sense. So at that point, on what grounds could you claim a higher sample size would be better? To get more exact confidence intervals if they are asymptotic?

Personally I would feel more convinced with a sample of 1000 rather than 500, but I can't identify technically where this makes a difference aside from precision.

• Andrew Gelman has discussed this extensively on his blog, e.g. here but there are many more posts on the topic. Aug 16 at 11:22
• Suppose you make a Type I error (which you expect to do $5\%$ or whatever of the time). With a small sample size, this is often associated with an apparently large effect. With a large sample size it it more likely to be associated with a small effect which you might decide is not substantial enough to pursue further; if the effect is both significant and substantial (with a narrow confidence interval, thanks to the large sample size) you are going to believe it more Aug 16 at 11:25
• I suppose it also matters what you mean by "small". ... Also, to what extent we trust the sampling method. If we encounter a medical study that uses eight patients, we might wonder if this represents an unbiased sample, or if there is some selection bias that led to these eight being used for the study. Aug 16 at 11:42
• @Henry If $\alpha=0.05$, you only make a type I error 5% of the time that H0 is true. If H0 is usually not true in the situations you deal with, you might almost never make a type I error. (The point you're trying to make should still apply) Aug 16 at 16:55
• "Statistically, if you have the sample size to detect an effect, then the sample size was large enough in that sense" - if I give 1 person a cholesterol drug and 1 person a placebo, and the former individual ends up with lower cholesterol, then I certainly detected an effect, but the sample size definitely isn't large enough to make generalisations about the population as a whole based on that. Or by "detect an effect", do you mean within a small confidence interval? If so, that would necessarily exclude studies with too small sample sizes, as this creates a large confidence interval. Aug 17 at 11:45

A comment made by "Josh" under this Andrew Gelman's blog post:

https://statmodeling.stat.columbia.edu/2022/01/24/how-large-a-sample-size-does-he-actually-need-he-got-statistical-significance-twice-isnt-that-enough/

helped me (finally!) understand this issue. I copied the comment here, hope that's OK:

"Let’s say we take 100 samples under the null distribution and 100 samples under the alternative distribution. We expect ~5 samples under the null to be significant. If we have power of 80 percent, we’d expect ~80 significant results under the alternative. Here only 5/85 of our results are false positives.

However, what if we have statistical power of only 10 percent. Then we expect ~10 samples under the alternative to be significant. Here 5/15 of our significant results would be false positives.

Decreasing power increases the likelihood that we are in situation one (significant result with the null) rather than in situation two (significant result with the alternative)."

• And, taking this to its extreme, a valid frequentist 5% level test is to collect no data, at all, and to reject the null hypothesis if a random number drawn from the Uniform(0, 1) distribution is $\leq 0.05$. This may seem like a silly example, where obviously the study result tells you nothing about the hypotheses you investigated. However, extremely underpowered studies are often barely better than this (as e.g. argued by Gelman here, but others before)). Aug 16 at 12:38
• Josh's comment correctly sets up Pr{Type I error} = Pr{unusual data | H0 is true} and Pr{Type II error} = 1 - Pr{unusual data | H1 is true} as long-run relative frequencies. But then it assumes -- without saying why -- that 50% of all hypotheses are null. Without Pr{H0 is true} we don't know what proportion of significant results are false positive. Actually we can be in situation two even though the power is high. Aug 16 at 21:10
• Thanks for the clarifications and corrections, @Björn and @dipetkov! Aug 18 at 7:29

Let's use reductio ad absurdum to see how the "if you have the sample size to detect an effect, then the sample size was large enough in that sense" logic doesn't hold. Imagine that you had a sample of size $$N=1$$. In such a sample, you can either observe a result that confirms or disagrees with your hypothesis. Unless you can make a strong hypothesis like "it should never happen" (but then, you don't need statistics, just logical reasoning), there will always be some chance to observe either of the results, even if unlikely. In such a case, your single sample does not "prove" anything, as it could be pure luck. However, if you repeated the experiment many times and saw the same result repeated, it gets less likely that it was luck.

• I'm not sure this quite address the question, it's basically stating the fact that for any non-zero effect size, the p-value tends to go down as the sample size goes up. But here we're starting with the assumption that the p-value is indeed significant even when small. Even when two studies return identical p-values, the one with larger sample size is more likely to have a significant p-value represent a true effect rather than a false positive. Higher sample size implies higher power and lower false discovery rate even with the same Type I error rate. Aug 16 at 20:57
• @NuclearHoagie ok, but the reasons are the same: with a larger sample, less chance is involved. I am not going into technical details on purpose, as other answers already did in, just showing the most basic intuition that disproves the main argument.
– Tim
Aug 17 at 6:27

In frequentist statistics this is a matter of $$\alpha$$ (significance level) and $$1-\beta$$ (power) of a hypothesis test. Usually you would fix $$\beta$$ (for ex. $$80 \%$$) and then determine the number of samples required to obtain such power for a desired $$\alpha$$ (for ex. $$0.05$$).

A specific example: power analysis for a simple t-test given $$d = 1$$, $$1-\beta=0.8$$, and $$\alpha=0.05$$ returns a required sample size of $$n \approx 17$$. Increasing $$1-\beta=0.9$$ yields $$n \approx 22$$. The significance level has stayed the same in both cases, i.e. the CI obtained from such tests will cover the true values with the same proportions, what has changed is the power, the second test has more power than the first (and narrower CI's).