How to scale the squares in a forest plot?
I'd argue for scaling the size (area) of the squares proportional to the weight that the study contributed to the meta-analysis. By scaling by weight, the area of the square is a direct visual cue of the relative impact a study had on the summary effect. The weight is among other things proportional to the standard error (precision) which, in turn, is usually (but not always!) directly related to the study sample size. This makes the most sense to me because the forest plot is a visual display of a statistical analysis which, in effect, is a weighted mean. Scaling by quality seems problematic to me because quality is difficult to measure objectively and the summary effect is not calculated using "quality-weights" (at least I've never seen it).
This seems to be supported by a number of authors. Steff Lewis and Mike Clarke$^{[1]}$ go into the history of the forest plot and write
The area of each square was proportional to the weight that the
individual study contributed to the meta analysis.
Michael Borenstein et al.$^{[2]}$ recommend the same when they explain
[...] the point is a box, proportional (in area) to that study’s
weight in the analysis.
This is again mirrored in Jonathan Sterne's book$^{[3]}$:
The size of the plotting symbol for the point estimate in each study is proportional to the weight that each trial contributes in the meta-analysis.
Lastly, in The Handbook of Research Synthesis and Meta-Analysis$^{[4]}$ we read
In this plot, each study is represented by a square, bounded on either side by a confidence interval. The location of each square on the horizontal axis represents the effect size for that study. The confidence interval represents the precision with which the effect size has been estimated, and the size of each square is proportional to the weight that will be assigned to the study when computing the combined effect.
A disadvantage of this approach becomes obvious when you want to present the results of a fixed-effects and a random-effects meta-analysis (or just two different analysis methods) in the same forest plot. Different analysis methods likely assign different weights to the studies and so the scaling of the area of the squares becomes ambiguous.
Estimators of between-study variance: Which one to use?
The metafor package for R offers no less than nine different estimators for the amount of heterogeneity:
- DerSimonian-Laird
- Hedges
- Hunter-Schmidt
- Sidik-Jonkman
- Maximum-likelihood
- Restricted maximum-likelihood (REML)
- Empirical Bayes
- Paule-Mandel
- Generalized Q-statistic
Descriptions of these estimators can be found in references $[4, 5, 6, 7, 8]$. The question remains which one to use? Veroniki et al.$^{[7]}$ and Langan et al.$^{[8]}$ recommend the Paule-Mandel estimator or restricted maximum likelihood based on simulation studies. A never publication by Langan et al.$^{[9]}$ made the following recommendations:
See also this question.
References
$[1]$ Lewis Steff, Clarke Mike. Forest plots: trying to see the wood and the trees BMJ. 2001. 322:1479 [link]
$[2]$ Michael Borenstein, Larry V. Hedges, Julian P.T. Higgins, Hannah R. Rothstein. Introduction to Meta-Analysis. Wiley 2009.
$[3]$ Jonathan Sterne. Meta-Analysis: An Updated Collection from the Stata Journal. Stata Press 2009.
$[4]$ Harris Cooper, Larry V. Hedges, Jeffrey C. Valentine (ed). The Handbook of Research Synthesis and Meta-Analysis. 2nd ed. Russell Sage Foundation 2009.
$[5]$ Rebecca DerSimonian, Raghu Kacker. Random-effects model for meta-analysis of clinical trials: An update. Contemp Clin Trials 28. 2007. 105-114. [link]
$[6]$ Wolfgang Viechtbauer, José Antonio López-López. A Comparison of Procedures to Test for Moderators in Mixed-Effects Meta-Regression Models. Psychological Methods 20(3). 2015. 360-374. [link]
$[7]$ Areti Angeliki Veroniki et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Syn Meth 7. 2016. 55-79. [link]
$[8]$ Dean Langan, Julian PT Higgins, Mark Simmonds. Comparative performance of heterogeneity variance estimators in meta-analysis: a review of simulation studies. Res Syn Meth 8. 2017. 181-198. [link]
$[9]$ Dean Langan et al. A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses. Res Syn Meth 10. 2019. 83-98. [link]
