3
$\begingroup$

In most meta-analyses that I've read, the authors choose to exclude studies with really small sample sizes (e.g n=10). Why is that?

Speculation

I could speculate that one reason is that since the effect size from a study depends on the means of the groups as well as the pooled standard deviation, like so

$$\frac{\bar{X}_1-\bar{X}_2}{SD_{pooled}}$$

and small samples cannot be assumed to achieve enough of a normal distribution among their results, the pooled standard deviation will be terribly off, thereby possibly messing up the overall effect size.

Further, some authors argue that studies with large sample sizes, among other things, tend to be of higher quality than studies with smaller sample sizes, and that there might be a publication bias going on, with smaller studies having to have positive results to get published. For example, this is the argument laid forward by Nüesch et al., 2010, although small studies are here defined as $n<100$. However, I guess one could extrapolate from results like these to what to do with even smaller studies.

References

  • Nüesch et al., 2010. Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study. BMJ, 2010 July 16, 10.1136/bmj.c3515.
$\endgroup$
  • 4
    $\begingroup$ To the best of my knowledge, there is no proven 'mathematical' argument for excluding 'small' studies from a meta-analysis as they will get generally get a 'small' amount of weight. The main issue is really with the workload of extracting and managing data, conducting critical analysis, etc. on studies that are deemed to provide very little input into the final assessment. This is contradictory to the principles of systematically reviewing the evidence, but a meta-analysis does not particularly need to be get it's data from a systematic review. $\endgroup$ – abousetta Apr 3 '14 at 9:22
3
$\begingroup$

If the meta-analysis uses fixed effects then small studies, other things being equal, will get very little weight and so the substantive conclusions are unlikely to be affected by including or excluding them.

If the meta-analysis uses random effects and there is substantial heterogeneity then the weights tend to become more equal. In that case even a small study may have almost the same weight as a large one. Some people feel this is undesirable because they believe that (a) small studies are of poorer quality (b) those small studies which they can find are not likely to be a random sample of all the small studies there have ever been.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.