# Pros and cons of meta-analyses

I have been considering doing some meta-analysis for a particular field of study in evolution, but before I go any further I would like to know; what are the positives and negatives of the process? For example, no need for a practical experiment is an advantage (time & money) but there will be a publication bias (more exciting results get published) which would be a disadvantage.

What papers in statistics journals discuss the pros and cons of meta-analysis?

• Cons: (i) Publication bias; journals reject insignificant findings (ii) File drawer effect; researchers stopping trials that are starting out badly or withholding null result trials (iii) Foreign journal bias; insignificant results get shoved into foreign journals, which don't get counted in meta-analyses as much. This is especially a problem in parapsychology. A way to possibly fix this is to create a central body that records the intention to experiment preceding the commencement of the experiment, then meta-analyses only consider those that recorded their prior intention with this body. – user14281 Nov 11 '12 at 11:02
• What might interest you is Rosenthal's $N$, which is a statistic that tells you how many null-result studies would have to have been file-drawed for the observed significance to disappear. – user14281 Nov 11 '12 at 11:04

Introduction to Meta-Analysis by Borenstein, Hedges, Higgins and Rothstein provides a detailed discussion of the pros and cons of meta-analysis. See for example the chapter "Criticism of meta-analysis" where the authors respond to various criticisms of meta-analysis. I note the section headings for that chapter and then make a few observations from memory that relate to that point:

• "one number cannot summarise a research field": A good meta analysis will model variability in true effect sizes and model the uncertainty of estimates.
• "the file drawer problem invalidates meta-analysis": Funnel plots and related tools allows you to see whether sample size is related to effect size in order to check for publication bias. Good meta-analyses endeavour to obtain unpublished studies. This issue is shared with narrative studies.
• "Mixing apples and oranges": Good meta-analyses provide a rigorous coding system for categorising included studies and justifying the inclusion and exclusion of studies in the meta-analysis. After studies have been classified, moderator analysis can be performed to see whether effect sizes vary across study type.
• "Important studies are ignored": You can code for the evaluated quality of the studies. Large samples can be given greater weighting.
• "meta analysis can disagree with randomised trials":
• "meta-analyses are performed poorly": This is merely an argument for improving the standards of meta-analytic methods.
• "Is a narrative review better?": Many of the critiques of meta-analysis (e.g., publication bias) are shared by narrative reviews. It is just that the methods of inference are less explicit and less rigorous in narrative reviews.

In my experience doing them, if they haven't been done before, as in you're not providing your own twist on an area, then the right journals don't have bias against them. A meta-analysis won't get in Science but in your field good journals are usually fine with new meta-analyses.

The time and cost saved not doing an experiment is often eaten up doing other things. One of the biggies is that many articles don't report sufficient information to analyze. You often have to contact the authors to recover this and they unfortunately all to frequently either cannot or will not comply with requests. It's the biggest time sink of the process.

You also missed out some pros like high citation rates. If you are the first and only meta-analysis new researchers will very often cite your paper. Another pro is relatively easy followup studies. In a year or two, in a dynamic field of study, you simply have to add the next two years of research to followup meta-analyses. It's relatively easy to co-opt meta-analyses in an area of study if you're the first mover. It then leads to relatively high citation rates.

If you're concerned that the results you're retrieving from the literature have publication bias there are statistical techniques such as funnel plots (study size (often -se) on the y-axis and effect on the x) that can be used to detect such. An unbiased literature on a subject will tend to have results that are symmetric in a funnel plot but an effect due to publication bias will look much more like it's just half of a distribution. And unlike doing experiments, finding that the data going into a meta-analysis is biased is publishable.

• My first thought about the publication bias was that the OP is concerned about the data accessible via literature study, not whether about how to publish the results of the meta-analysis. – cbeleites Nov 8 '12 at 23:43
• Yes I was thinking more towards strengths and weaknesses which I should consider when deciding whether or not to do it, and so I can minimise their effects. – rg255 Nov 9 '12 at 9:32

I thought I would do a criticism of the "Criticism of meta-analysis" with apologies to Michael Borenstein and colleagues.

• "one number cannot summarise a research field": A good meta analysis will model variability in true effect sizes and model the uncertainty of estimates.

! Variance is just another possibly misleading summary as is unceartainty and both will be very misleading if biases that are almost surely there are not explicitly dealt with.

• "the file drawer problem invalidates meta-analysis": Funnel plots and related tools allows you to see whether sample size is related to effect size in order to check for publication bias. Good meta-analyses endeavour to obtain unpublished studies. This issue is shared with narrative studies.

! As Box once said - like sending out a row boat to see if the seas are calm enough for the Queen Mary to travel into. Very low power and almost surely mis-specified censoring process.

• "Mixing apples and oranges": Good meta-analyses provide a rigorous coding system for categorising included studies and justifying the inclusion and exclusion of studies in the meta-analysis. After studies have been classified, moderator analysis can be performed to see whether effect sizes vary across study type.

! Again hopeless power and usually agregation bias as well.

• "Important studies are ignored": You can code for the evaluated quality of the studies. Large samples can be given greater weighting.

! Now hopeless power, model mis-specification and bias not always properly accounted for see On the bias produced by quality scores in meta-analysis

• "meta analysis can disagree with randomised trials":

! Fully agree and also the only source about the real uncertainty of them.

• "meta-analyses are performed poorly": This is merely an argument for improving the standards of meta-analytic methods.

! Fully agree.

• "Is a narrative review better?": Many of the critiques of meta-analysis (e.g., publication bias) are shared by narrative reviews. It is just that the methods of inference are less explicit and less rigorous in narrative reviews.

! Fully agree.

Not sure why much of the meta-analysis literature maintians such rose coloured glasses - meta-analyses have to be done Meta-analysis in medical research: Strong encouragement for higher quality in individual research efforts, but should be critically done with full awareness of all the worts.

And, as I almost always forget, I need to clarify what exactly I mean by meta-analysis as what others take it to mean has varied over time and place and perhaps the most common meaning today - just the quantitative methods used on extracted numbers obtained in a systematic review - is not what I mean. I mean the whole systematic review process even if it is decided not to actually use any quantitative methods at all. Or in just one sentence as quoted in wiki

In statistics, a meta-analysis refers to methods focused on contrasting and combining results from different studies, in the hope of identifying patterns among study results, sources of disagreement among those results, or other interesting relationships that may come to light in the context of multiple studies.

• Good point, narrative reviews give more freedom to discuss the strengths and weaknesses of previous studies, perhaps meta-analyses should take on a more narrative role and discuss the existing studies more rather than try to draw new conclusions from old (probably biased and variable quality) data. – rg255 Nov 16 '12 at 10:17
• @rg255 I have added a bit at the end to address your comment. Also perhaps the conclusions from the Strong encouragement reference would be relevant. – phaneron Nov 16 '12 at 16:17