I thought I would do a criticism of the "Criticism of meta-analysis" with apologies to Michael Borenstein and colleagues.
- "one number cannot summarise a research field": A good meta analysis will model variability in true effect sizes and model the uncertainty of estimates.
! Variance is just another possibly misleading summary as is
unceartainty and both will be very misleading if biases that are almost
surely there are not explicitly dealt with.
- "the file drawer problem invalidates meta-analysis": Funnel plots and related tools allows you to see whether sample size is related to effect size in order to check for publication bias. Good meta-analyses endeavour to obtain unpublished studies. This issue is shared with narrative studies.
! As Box once said - like sending out a row boat to see if the seas
are calm enough for the Queen Mary to travel into. Very low power and
almost surely mis-specified censoring process.
- "Mixing apples and oranges": Good meta-analyses provide a rigorous coding system for categorising included studies and justifying the inclusion and exclusion of studies in the meta-analysis. After studies have been classified, moderator analysis can be performed to see whether effect sizes vary across study type.
! Again hopeless power and usually agregation bias as
- "Important studies are ignored": You can code for the evaluated quality of the studies. Large samples can be given greater weighting.
! Now hopeless power, model mis-specification and bias not always
properly accounted for see On the bias produced by quality scores in meta-analysis
- "meta analysis can disagree with randomised trials":
agree and also the only source about the real uncertainty of them.
- "meta-analyses are performed poorly": This is merely an argument for improving the standards of meta-analytic methods.
- "Is a narrative review better?": Many of the critiques of meta-analysis (e.g., publication bias) are shared by narrative reviews. It is just that the methods of inference are less explicit and less rigorous in narrative reviews.
Not sure why much of the meta-analysis literature maintians such rose coloured glasses - meta-analyses have to be done Meta-analysis in medical research: Strong encouragement for higher quality in individual research efforts, but should be critically done with full awareness of all the worts.
And, as I almost always forget, I need to clarify what exactly I mean by meta-analysis as what others take it to mean has varied over time and place and perhaps the most common meaning today - just the quantitative methods used on extracted numbers obtained in a systematic review - is not what I mean. I mean the whole systematic review process even if it is decided not to actually use any quantitative methods at all. Or in just one sentence as quoted in wiki
In statistics, a meta-analysis refers to methods focused on
contrasting and combining results from different studies, in the hope
of identifying patterns among study results, sources of disagreement
among those results, or other interesting relationships that may come
to light in the context of multiple studies.