Pros and cons of meta-analyses

Question

I have been considering doing some meta-analysis for a particular field of study in evolution, but before I go any further I would like to know; what are the positives and negatives of the process? For example, no need for a practical experiment is an advantage (time & money) but there will be a publication bias (more exciting results get published) which would be a disadvantage.

What papers in statistics journals discuss the pros and cons of meta-analysis?

Answer 1

Introduction to Meta-Analysis by Borenstein, Hedges, Higgins and Rothstein provides a detailed discussion of the pros and cons of meta-analysis. See for example the chapter "Criticism of meta-analysis" where the authors respond to various criticisms of meta-analysis. I note the section headings for that chapter and then make a few observations from memory that relate to that point:

• "one number cannot summarise a research field": A good meta analysis will model variability in true effect sizes and model the uncertainty of estimates.
• "the file drawer problem invalidates meta-analysis": Funnel plots and related tools allows you to see whether sample size is related to effect size in order to check for publication bias. Good meta-analyses endeavour to obtain unpublished studies. This issue is shared with narrative studies.
• "Mixing apples and oranges": Good meta-analyses provide a rigorous coding system for categorising included studies and justifying the inclusion and exclusion of studies in the meta-analysis. After studies have been classified, moderator analysis can be performed to see whether effect sizes vary across study type.
• "Important studies are ignored": You can code for the evaluated quality of the studies. Large samples can be given greater weighting.
• "meta analysis can disagree with randomised trials":
• "meta-analyses are performed poorly": This is merely an argument for improving the standards of meta-analytic methods.
• "Is a narrative review better?": Many of the critiques of meta-analysis (e.g., publication bias) are shared by narrative reviews. It is just that the methods of inference are less explicit and less rigorous in narrative reviews.

Answer 2

In my experience doing them, if they haven't been done before, as in you're not providing your own twist on an area, then the right journals don't have bias against them. A meta-analysis won't get in Science but in your field good journals are usually fine with new meta-analyses.

The time and cost saved not doing an experiment is often eaten up doing other things. One of the biggies is that many articles don't report sufficient information to analyze. You often have to contact the authors to recover this and they unfortunately all to frequently either cannot or will not comply with requests. It's the biggest time sink of the process.

You also missed out some pros like high citation rates. If you are the first and only meta-analysis new researchers will very often cite your paper. Another pro is relatively easy followup studies. In a year or two, in a dynamic field of study, you simply have to add the next two years of research to followup meta-analyses. It's relatively easy to co-opt meta-analyses in an area of study if you're the first mover. It then leads to relatively high citation rates.

If you're concerned that the results you're retrieving from the literature have publication bias there are statistical techniques such as funnel plots (study size (often -se) on the y-axis and effect on the x) that can be used to detect such. An unbiased literature on a subject will tend to have results that are symmetric in a funnel plot but an effect due to publication bias will look much more like it's just half of a distribution. And unlike doing experiments, finding that the data going into a meta-analysis is biased is publishable.

Answer 3

I thought I would do a criticism of the "Criticism of meta-analysis" with apologies to Michael Borenstein and colleagues.

• "one number cannot summarise a research field": A good meta analysis will model variability in true effect sizes and model the uncertainty of estimates.

! Variance is just another possibly misleading summary as is unceartainty and both will be very misleading if biases that are almost surely there are not explicitly dealt with.

• "the file drawer problem invalidates meta-analysis": Funnel plots and related tools allows you to see whether sample size is related to effect size in order to check for publication bias. Good meta-analyses endeavour to obtain unpublished studies. This issue is shared with narrative studies.

! As Box once said - like sending out a row boat to see if the seas are calm enough for the Queen Mary to travel into. Very low power and almost surely mis-specified censoring process.

• "Mixing apples and oranges": Good meta-analyses provide a rigorous coding system for categorising included studies and justifying the inclusion and exclusion of studies in the meta-analysis. After studies have been classified, moderator analysis can be performed to see whether effect sizes vary across study type.

! Again hopeless power and usually agregation bias as well.

• "Important studies are ignored": You can code for the evaluated quality of the studies. Large samples can be given greater weighting.

! Now hopeless power, model mis-specification and bias not always properly accounted for see On the bias produced by quality scores in meta-analysis

• "meta analysis can disagree with randomised trials":

! Fully agree and also the only source about the real uncertainty of them.

• "meta-analyses are performed poorly": This is merely an argument for improving the standards of meta-analytic methods.

! Fully agree.

• "Is a narrative review better?": Many of the critiques of meta-analysis (e.g., publication bias) are shared by narrative reviews. It is just that the methods of inference are less explicit and less rigorous in narrative reviews.

! Fully agree.

Not sure why much of the meta-analysis literature maintians such rose coloured glasses - meta-analyses have to be done Meta-analysis in medical research: Strong encouragement for higher quality in individual research efforts, but should be critically done with full awareness of all the worts.

And, as I almost always forget, I need to clarify what exactly I mean by meta-analysis as what others take it to mean has varied over time and place and perhaps the most common meaning today - just the quantitative methods used on extracted numbers obtained in a systematic review - is not what I mean. I mean the whole systematic review process even if it is decided not to actually use any quantitative methods at all. Or in just one sentence as quoted in wiki

In statistics, a meta-analysis refers to methods focused on contrasting and combining results from different studies, in the hope of identifying patterns among study results, sources of disagreement among those results, or other interesting relationships that may come to light in the context of multiple studies.