10
$\begingroup$

I'm asking myself the question of why studies with small sample sizes are not as convincing as those with larger sample sizes, and when this becomes a statistical issue. A complaint I've heard a lot with studies is while they show a certain "desired" result (e.g. the treatment group had lower cholesterol than the placebo group), the sample size was small.

Statistically, if you have the sample size to detect an effect, then the sample size was large enough in that sense. So at that point, on what grounds could you claim a higher sample size would be better? To get more exact confidence intervals if they are asymptotic?

Personally I would feel more convinced with a sample of 1000 rather than 500, but I can't identify technically where this makes a difference aside from precision.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Andrew Gelman has discussed this extensively on his blog, e.g. here but there are many more posts on the topic. $\endgroup$
    – Richard Hardy
    Aug 16, 2022 at 11:22
  • 2
    $\begingroup$ Suppose you make a Type I error (which you expect to do $5\%$ or whatever of the time). With a small sample size, this is often associated with an apparently large effect. With a large sample size it it more likely to be associated with a small effect which you might decide is not substantial enough to pursue further; if the effect is both significant and substantial (with a narrow confidence interval, thanks to the large sample size) you are going to believe it more $\endgroup$
    – Henry
    Aug 16, 2022 at 11:25
  • $\begingroup$ I suppose it also matters what you mean by "small". ... Also, to what extent we trust the sampling method. If we encounter a medical study that uses eight patients, we might wonder if this represents an unbiased sample, or if there is some selection bias that led to these eight being used for the study. $\endgroup$
    – Sal Mangiafico
    Aug 16, 2022 at 11:42
  • 2
    $\begingroup$ @Henry If $\alpha=0.05$, you only make a type I error 5% of the time that H0 is true. If H0 is usually not true in the situations you deal with, you might almost never make a type I error. (The point you're trying to make should still apply) $\endgroup$
    – Glen_b
    Aug 16, 2022 at 16:55
  • $\begingroup$ "Statistically, if you have the sample size to detect an effect, then the sample size was large enough in that sense" - if I give 1 person a cholesterol drug and 1 person a placebo, and the former individual ends up with lower cholesterol, then I certainly detected an effect, but the sample size definitely isn't large enough to make generalisations about the population as a whole based on that. Or by "detect an effect", do you mean within a small confidence interval? If so, that would necessarily exclude studies with too small sample sizes, as this creates a large confidence interval. $\endgroup$
    – NotThatGuy
    Aug 17, 2022 at 11:45

3 Answers 3

Reset to default
10
$\begingroup$

A comment made by "Josh" under this Andrew Gelman's blog post:

https://statmodeling.stat.columbia.edu/2022/01/24/how-large-a-sample-size-does-he-actually-need-he-got-statistical-significance-twice-isnt-that-enough/

helped me (finally!) understand this issue. I copied the comment here, hope that's OK:

"Let’s say we take 100 samples under the null distribution and 100 samples under the alternative distribution. We expect ~5 samples under the null to be significant. If we have power of 80 percent, we’d expect ~80 significant results under the alternative. Here only 5/85 of our results are false positives.

However, what if we have statistical power of only 10 percent. Then we expect ~10 samples under the alternative to be significant. Here 5/15 of our significant results would be false positives.

Decreasing power increases the likelihood that we are in situation one (significant result with the null) rather than in situation two (significant result with the alternative)."

Improve this answer
$\endgroup$
3
  • 7
    $\begingroup$ And, taking this to its extreme, a valid frequentist 5% level test is to collect no data, at all, and to reject the null hypothesis if a random number drawn from the Uniform(0, 1) distribution is $\leq 0.05$. This may seem like a silly example, where obviously the study result tells you nothing about the hypotheses you investigated. However, extremely underpowered studies are often barely better than this (as e.g. argued by Gelman here, but others before)). $\endgroup$
    – Björn
    Aug 16, 2022 at 12:38
  • 1
    $\begingroup$ Josh's comment correctly sets up Pr{Type I error} = Pr{unusual data | H0 is true} and Pr{Type II error} = 1 - Pr{unusual data | H1 is true} as long-run relative frequencies. But then it assumes -- without saying why -- that 50% of all hypotheses are null. Without Pr{H0 is true} we don't know what proportion of significant results are false positive. Actually we can be in situation two even though the power is high. $\endgroup$
    – dipetkov
    Aug 16, 2022 at 21:10
  • $\begingroup$ Thanks for the clarifications and corrections, @Björn and @dipetkov! $\endgroup$
    – Sointu
    Aug 18, 2022 at 7:29
5
$\begingroup$

Let's use reductio ad absurdum to see how the "if you have the sample size to detect an effect, then the sample size was large enough in that sense" logic doesn't hold. Imagine that you had a sample of size $N=1$. In such a sample, you can either observe a result that confirms or disagrees with your hypothesis. Unless you can make a strong hypothesis like "it should never happen" (but then, you don't need statistics, just logical reasoning), there will always be some chance to observe either of the results, even if unlikely. In such a case, your single sample does not "prove" anything, as it could be pure luck. However, if you repeated the experiment many times and saw the same result repeated, it gets less likely that it was luck.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ I'm not sure this quite address the question, it's basically stating the fact that for any non-zero effect size, the p-value tends to go down as the sample size goes up. But here we're starting with the assumption that the p-value is indeed significant even when small. Even when two studies return identical p-values, the one with larger sample size is more likely to have a significant p-value represent a true effect rather than a false positive. Higher sample size implies higher power and lower false discovery rate even with the same Type I error rate. $\endgroup$
    – Nuclear Hoagie
    Aug 16, 2022 at 20:57
  • $\begingroup$ @NuclearHoagie ok, but the reasons are the same: with a larger sample, less chance is involved. I am not going into technical details on purpose, as other answers already did in, just showing the most basic intuition that disproves the main argument. $\endgroup$
    – Tim ♦
    Aug 17, 2022 at 6:27
4
$\begingroup$

In frequentist statistics this is a matter of $\alpha$ (significance level) and $1-\beta$ (power) of a hypothesis test. Usually you would fix $\beta$ (for ex. $80 \%$) and then determine the number of samples required to obtain such power for a desired $\alpha$ (for ex. $0.05$).

A specific example: power analysis for a simple t-test given $d = 1$, $1-\beta=0.8$, and $\alpha=0.05$ returns a required sample size of $n \approx 17$. Increasing $1-\beta=0.9$ yields $n \approx 22$. The significance level has stayed the same in both cases, i.e. the CI obtained from such tests will cover the true values with the same proportions, what has changed is the power, the second test has more power than the first (and narrower CI's).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.