Moderator analysis in multilevel meta-analysis of dependent effect sizes

Question

I'm conducting a meta-analysis with a data set that is somewhat similar in structure to this post, where there are multiple effect sizes per study. Specifically, most studies I'm analyzing contain multiple experiments, and the effect sizes from these experiments are dependent (i.e., are derived from the same participants). Here's a depiction of what a small part the data looks like:

esid  studyid  sampleid  testtype
   1        1         1         a
   2        2         2         b
   3        2         3         b
   4        3         4         a
   5        3         4         a
   6        3         4         b        

• esid = effect size identifier
• studyid = study identifier (same number = effect size from same study)
• sampleid = sample (correlated effect) identifier (same number = effect size from same participants)

I'm planning on using metafor to fit a three-level model (similar to Konstantopoulos, 2011), including a random effect at the studyid level (i.e., paper level) since effect sizes are nested within studies:

ml.mod <- rma.mv(yi, vi, random = ~1 | studyid/esid, data = dat)

Then, using the robust function to account for the dependency between some effect sizes (clustering at the sampleid level):

robust(ml.mod, cluster = dat$sampleid, adjust=TRUE)

As far as I can tell, this is basically what Dr. Viechtbauer suggests here. Assuming this approach looks reasonable to deal with this data structure (and please, let me know if it doesn't!), my question has to do with how to best investigate whether testtype influences the estimate. Because testtype varies both within- and between-studies (i.e., some studies use both testtype a and b - and these estimates are dependent - others only a, others only b), is there a best approach to dealing with this question? For example, would it be reasonable to directly compare the testtypes:

ml.mod <- rma.mv(yi, vi, mods = cbind(testtype), random = ~1 | studyid/esid, data = dat)
robust(ml.pub, cluster = dat$sampleid, adjust = TRUE)

Or would it be more appropriate to conduct separate meta-analyses looking at the effect when using testtype a and then separately using testtype b (and not directly compare them):

ml.mod.a <- rma.mv(yi, vi, random = ~1 | studyid/esid, data = dat, subset=(testtype=="a")
robust(ml.mod.a, cluster = dat$sampleid, adjust = TRUE)

ml.mod.b <- rma.mv(yi, vi, random = ~1 | studyid/esid, data = dat, subset=(testtype=="b")
robust(ml.mod.b, cluster = dat$sampleid, adjust = TRUE)

Any suggestions would be extremely appreciated by this newbie! Thank you!

Answer 1

With respect to the clustering:

For the cluster-robust method, you should not cluster at the sampleid level, as that would treat the second and third effect as independent, even though they come from the same study (although different samples). You should cluster at the outermost grouping level, which would be studyid. So use:

robust(ml.mod, cluster = dat$studyid, adjust=TRUE)

As for the main question:

The standard approach would be to just include testtype as a moderator. But please use the formula syntax, so use:

ml.mod <- rma.mv(yi, vi, mods = ~ testtype, random = ~1 | studyid/esid, data = dat)

As you correctly point out, you have a mix of between- and within-study evidence about the difference between levels a and b of the testtype moderator. Generally speaking, between-study evidence is weaker, since it is more likely to be confounded by other variables that vary between studies. Therefore, besides the model above, I would suggest to also examine what happens when you only include studies that provide within-study evidence. So, create a dataset that is a subset of dat that only includes those studies that include both levels of testtype. Then compare the results and see if conclusions are consistent.