# Metafor package: bias and sensitivity diagnostics

I am conducting a multi-level meta-analysis that includes some articles with multiple outcomes. Therefore I am using the rma.mv() function. Example code:

test.main = rma.mv(yi,vi,random = ~1|ID, data = data)


I have two questions:

1. I read in a previous query that when using rma.mv(), ranktest() is not a reliable test of funnel plot asymmetry. However, if sample variance was added to the original model as a moderator, then this model would be similar to the Egger's test:

test.egger = rma.mv(yi,vi, mod = vi, random = ~1|ID, data = data)


Is this code a correct interpretation of that guidance? Also, are funnel plots also (more or less) useless as a tool with rma.mv() models?

2. Neither leave1out() nor trimfill() work with rma.mv() to assess sensitivity of model results. Are other sensitivity analysis tools currently available for rma.mv() models that do not involve a masterful understanding of R?

Regarding 1: Yes, adding vi as a moderator is indeed the logical way of extending Egger's test to more complex models.

Actually, using the sampling variances as a moderator is just one possibility of conducting the "regression test for funnel plot asymmetry". Others have suggested using the inverse of the sampling variances or the standard errors (square-root of the sampling variances) or their inverses or the total sample sizes (or some function thereof) as moderators. It's not entirely clear which predictor is the "best" choice (and this can depend on what outcome measure you use for your meta-analysis). For example, for some measures, the equation we use to approximate/estimate the sampling variance is actually a function of the observed outcome, which automatically creates a relationship between the two even in the absence of publication bias (or "small-study bias" or whatever we want to call it). In that case, using just the sample sizes (or some function thereof) may be preferable.

But the main point is: Yes, the regression test can be easily applied when working with more complex models by adding an appropriate moderator to the model.

Whether funnel plots are useful or not when there are multilevel/multivariate structures underlying the data is debatable. For example, sets of points may cluster together because of statistical dependencies (that are accounted for when using an appropriate multilevel/multivariate model), but in the funnel plot, the points are just that: a bunch of points. That makes the interpretation of funnel plots more difficult because (unless you take extra steps by using different colors or symbols) you cannot see those inherent dependencies -- not that most people (myself included) are any good at interpreting funnel plots even in the simplest cases (there is empirical research demonstrating that!).

Regarding 2: Yes, a bunch of post model fitting functions do not currently work with rma.mv model objects. I just haven't gotten around to implementing this and some of this will actually require some thinking. For example, leave1out() removes one study at a time -- in the univariate context, this is equivalent to removing each observed outcome at a time, but what about multilevel/multivariate data? Also remove each observed outcome at a time? Or remove sets of points? Or make different options available? With respect to trim-and-fill (leaving aside the question of how useful this method really is): Extending the method to multilevel/multivariate data would be worth writing an entire paper about.

So, it's great that you want to do sensitivity analyses, but as of right now, you will have to do some of this manually. Leave-one-out analyses are easily done with a simple for-loop and thinking carefully about what "one" is (e.g., each observed outcome, each cluster/study). You can do the regression test and maybe leave trim-and-fill be for now. Standardized residuals are available via rstandard(), so you can examine the data for potential outliers. You can get the hat values via hatvalues() (just the leverages along the diagonal or the entire hat matrix) which gives you an indication which points have a strong influence on the results. Another really useful measure in this context is Cook's distance, which you can obtain via cooks.distance(), also for rma.mv objects.

• Wow! Thank you for your thoughtful and thorough answers. I will attempt to figure out how to use the for-loop code, as you suggest. Jun 7, 2015 at 15:35
• Glad to hear you found this useful. The various model fitting functions including rma.mv() have a subset argument that you can easily use together with a for-loop to drop (sets of) observed outcomes. Jun 7, 2015 at 20:37
• Wolfgang, the product of your suggestions can be observed here: aobpla.oxfordjournals.org/content/7/… Thanks again! Dec 16, 2015 at 4:38