Are low sample studies redundant most of the time?
Should all studies have large sample sizes?
OTOH the low sample is sometimes an always present thing. Millions of people cannot be easily studied.
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Not necessarily, but in that case I would wait until a couple of low-sample studies were performed and they all showed similar results, i.e. the results would be replicable. $\endgroup$– user2974951Commented Nov 4, 2021 at 7:53
-
1$\begingroup$ Unless you have publication bias, which tends to be a bit more of a problem with small studies. E.g. really large clinical trial usually get published, they are too large and there's too many people involved that want to get publications for their CV, but small studies are much more commonly unreported - with clinical trials nowadays registries help (even if some - esp. universities & other public research institutions - keep violating the results reporting requirements). In areas without such transparency measures it's bound to be much more of a problem. $\endgroup$– BjörnCommented Nov 4, 2021 at 8:15
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
No, they aren't. Some effects are indeed strong enough to be reliably detected even with small sample sizes. For instance, here is some preliminary data that just came my way:
No, this will not be the entire publication. There will be more plots of other variables in the two groups - but they all paint the same picture. This effect is novel enough and scientifically important enough to publish even with the small sample the researchers do have.
Why was an effect so strong not found before? Simple: because nobody thought about looking at it. Advances in understanding of one disease on the one hand, and a biological dynamic on the other hand led some people who had an overview of both fields to think of a study to combine the two aspects. And this kind of low hanging "combination fruit" will not run out any time soon.
I agree with the comments that small studies are never persuasive on their own, and that they should always be replicated, and included in systematic reviews. And that publication bias is a thing. Nevertheless, small studies that are well motivated by the state of knowledge - as the one that yielded that plot above - are worth doing.
Also, there is the economic argument. Resources are scarce, and acquiring data is often expensive. We may be able to fund either 10 large scale studies, or 1000 smaller scale studies. Yes, the larger studies will yield more precise results. But 1000 less precise results may be worth as much or more than 10 more precise ones.
answered Nov 4, 2021 at 8:37
$\begingroup$
$\endgroup$
The sample size needed is entirely dependent on the signal:noise ratio and the degree of experimental control. One dietary study with 24 patients put in a metabolic chamber for one week, with precise measurements and using a crossover design provided more information than a 10,000 person observational dietary study.
There are ways to compute sample sizes needed for model reliability, as a function of the signal:noise ratio (e.g. true $R^2$). See this.
answered Nov 4, 2021 at 12:51