4
$\begingroup$

I have done a multi-level meta analysis with R, using metafor package. My questions relates to funnel plot asymmetry.

(1) If a regression test suggested funnel asymmetry (i.e. suggested a publication bias is present), Does this mean that my meta-analysis model is bad? and my results of meta-analysis are bad?

(2) What are other possible reasons, besides publication bias, that could lead to asymmetry in the funnel plots?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Asymmetry is a fact about the world, it does not make your model bad. $\endgroup$ – mdewey May 23 '16 at 9:22
8
$\begingroup$

There are several things that can lead to an asymmetric funnel plot (and I doubt my list is complete - there are presumably other reasons, too):

  1. Publication bias in its various forms (selective publication of studies, journals being more interested in significant results, less impressive results ending up in hard to find obscure journals, selective publication of results from a study etc.)
  2. Related but not quite the same: when analyses are not fully pre-specified the various choices made during analyses, which may bias results one way or another ("garden of forking paths")
  3. Within study biases (e.g. you look at the odds ratio for people having a medical event, but one treatment group had fewer drop-outs and thus, longer follow-up than the other one)
  4. The choice of effect scale (e.g. plotting odds/risk/hazard ratios on a linear scale instead of a log-scale could make things look asymmetric, using odds ratios across studies of different length when there is a constant hazard ratio etc.)
  5. Chance (there are so many meta-analyses done that some will show some asymmetry just by chance - and this is to some extent part of the scientific process: things that work well - even if this is by chance - continue to be studied)
  6. When none of the above apply (which we cannot really know for sure): if you are plotting estimates from a model, then an asymmetry could still arise due to an issue with the model (as you mention).
|cite|improve this answer
$\endgroup$
  • $\begingroup$ May asymmetry arise from lack of moderators in the model as well? $\endgroup$ – Amer May 23 '16 at 7:20
  • 2
    $\begingroup$ I assume you are talking about non-randomized trials? If so, if different studies in the meta-analysis is biased to different degrees (e.g. they all omit the same moderators of the effect, but it matters to different degrees e.g. due to differences in study populations, or they are identical in design/population, but omit different moderators), then yes. These are varying levels of within study biases across studies (if you had the exact same bias in all trials, then of course the funnel plot could look perfectly symmetric given sufficiently many studies, but be centred on a biased effect). $\endgroup$ – Björn May 23 '16 at 7:48
  • 1
    $\begingroup$ There need not be any bias at all. If there is a relevant study-level moderator that has an impact on the size of the effect and it also happens to be correlated with sample size (e.g., the more invasive but more effective version of the treatment has only been applied in some smaller trials), then not including the moderator in your model can lead to the false conclusion that there is publication bias. $\endgroup$ – Wolfgang May 23 '16 at 8:07
  • 2
    $\begingroup$ But just to add: While I wholeheartedly agree that it is useful to examine whether there is a relationship between the effect size estimates and the corresponding sample sizes (or standard errors) or some function thereof, a relationship that is found doesn't have to be due to publication bias, shady researcher practices, or other biases -- there could very well be a rather innocuous explanation (that may even lead to insights). $\endgroup$ – Wolfgang May 23 '16 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.