Based on the answers I've read to a question1 on the Skeptics stack exchange, I began to wonder what methods besides the declaration of industry funding could be used to determine the validity of results from meta analysis?

In other words, can we only trust meta analysis if there is no apparent conflict of interest concerning a topic, or are there other methods?

1. Does the artificial sweetener aspartame cause cancer?

  • $\begingroup$ There are many resources on how to appraise a meta-analysis. Best starting point is probably the Oxman-Guyatt Index (but follow citing papers as it is an old contribution). $\endgroup$ Jul 8 '19 at 6:34
  • $\begingroup$ A valid meta analysis is intended to produce truth. However, if you have suspectful result, different approaches are available for a cross-validation of results of a meta-analysis. Furthermore, to substantiate a relationship etc, you can design an experiment and execute it. $\endgroup$ Jul 11 '19 at 8:43

There are many methods, mostly falling under the category "try to understand exactly what they did and use your best judgement".

Some stuff I would look into:

1) What were the inclusion and exclusion criteria for studies? Are they sensible? To what extent was subjective judgement involved in selecting the studies? How was it handled? Usually metaanalysis will explicitly state a search strategy - if you try to replicate it, will you find papers that were not included but look relevant?

2) Are they open about their input data? Can you get to the list of studies? Can you get the list of studies that matched the search but were excluded? Is the input data for the statistical method they used available? (either individual-level data or summary of effects from the individual studies) Are the input studies truly from various sources or is there an author/research group/company that supplied disproportionate part of the underlying studies? If so, is this author/... credible?

3) Is the statistical method clearly described? What are its assumptions? Are those likely to hold? Do they account for between-study differences in a sensible way? Does it look like they didn't report some analysis that would be obvious/straightforward to use? Plus points if multiple ways to analyze the data are reported and the differences in their conclusions are honestly discussed.

4) Is code for the statistical analysis available? If so, can you rerun their analyses? (this would likely be a pretty high bar to cross) If so, do you get the same results? What happens if you fiddle with the meta parameters of their model?

5) Is there a plot of all the data included? Does it look like it is consistent with the conclusions of the paper? Is the result driven by a single/few large study/studies dwarfing all the other included? If so, the metaanalysis is only as good as those big studies - check if they are trustworthy.

6) Do claims in the abstract match what is written in the "Conclusions" section? What are the stated limitations? (beware a study that does not state limitations or only states some obvious boileplate - real limitations are always there)

7) If claims are made about a broader population is the population well represented in the underlying studies? (e.g. beware of sweeping generalizations from only U.S. studies - unless you only care about the U.S.).

8) Check numerical measures of publication bias in the set of studies included - e.g. P-curve but see also this commentary by Andrew Gelman

Note that basically no metaanalysis would meet all of the criteria above (because science is hard). So you should be careful to not dismiss an analysis just because you can find a problem - you will always be able to find a problem. Instead, you should ask whether the problems you found are important for the question at hand and what can you learn from the analysis despite the problems.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.