I fitted the following hierarchical mixed-effects meta-analytic model using rma.mv() in the metafor R package.

mod<-rma.mv(yi=yi, V=vi, mods=~ region + subtype, random = ~ 1 |study/id, data = es, method = 'ML')

where yi are relative risks, V study variances, and region and subtype are dummy variables.

I then requested a funnel plot:


which looks like this:

funnel plot

However, because some articles in my meta-analysis use the same data (please see @Wolfgang reply here):


I also decided to run the fitted model through the robust() function, specifying study as a cluster effect:

mod_rob<-robust(mod, cluster=es$study)

I then requested a funnel plot of this model corrected for cluster effects:


Funnel plot after running model through robust()

The corresponding funnel plot looks really weird after correcting for cluster effects using the robust() function, with the data points aligned in two lines either side of the plot. I would be most grateful if anybody knows why this is and whether it is valid to use funnel plots in such instances? Thank you kindly.


1 Answer 1


This is an artifact of the way the cluster-robust method works (where a small residual leads to a small standard error for a particular study). However, those study-specific standard errors are not trustworthy (cluster-robust inferences work for the fixed effects of the model, not for individual data points). So, no, it is not valid to use funnel plots this way.

  • $\begingroup$ thank you very much indeed. I guess a follow up question is, how would you then recommend assessing for small study effects, potential publication bias etc. if the funnel plot should not be used after using using cluster-robust inference? Is the pseudo- Egger regression test still valid, that is to my understanding, fitting the original meta regression model with e.g. the study sample variances as an additional moderator and testing the null that the intercept is zero? Any other options? Thank you. $\endgroup$ Commented Apr 11, 2020 at 8:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.