My dataset

I have a dataset where Effect sizes are nested in Testing occasions, which are nested in Samples of participants, which in turn are nested in Studies (that is, Effect sizesTesting occasionsSamples of participantsStudies). In addition, I have several effect size-level moderators (that is, variables that can differ between different Effect sizes for the same Testing occasion) that I want to investigate, for example Type of measure.

It should be said that the dataset is quite chaotic, in that each study looks different when it comes to how many samples of participants it has, how many testing occasions each sample has, and how many effect sizes are included in each sample, what factors of certain moderators that are included, et cetera.

As an example of this high entropy, this is how just some of the data points might look (in the full data set, there are in total about 3600 data points and a lot more variables):

Study   Sample  Testing_occasion    Type_of_measure    Es        Var
1       1       1                   1                  -0.3787   0.2233
1       1       2                   1                  -0.1344   0.2202
2       1       1                   2                  0.02286   0.0400
3       1       1                   1                  -0.3796   0.0017
3       1       1                   3                  -0.3983   0.0017
3       2       1                   4                  -0.3977   0.0016
3       2       2                   1                  -0.4398   0.0016
3       3       1                   1                  -0.4021   0.0017
3       3       1                   2                  -0.4158   0.0017

What I've already tried

I've played around with the data and tried many different things, but I'm still not quite sure what the best way to proceed is. Here are some of the things I've tried:

  1. I've tried running a four-level meta-analysis using the rma.mv function from the Metafor package for R. However, since I have around 3600 individual effect sizes in my data set, it simply takes to long time to run these (I've been waiting multiple hours for just one analysis, without any moderators included at all, to finish without any luck). So this doesn't seem feasible (also, see concern regarding dependencies in analysis number 2).
  2. I've tried shifting the unit of analysis (see pp. 15-16 of Cooper 2006 for a concrete example), creating grand means on the sample level according to the factors of the moderator that I'm currently interested in (so that I, for example, get two effect sizes for a certain sample if two factors of the moderator in question are represented among the effect sizes) and then I apply a three-level meta-analysis (New effect sizesSamples of participantsStudies), again using the rma.mv function from the Metafor package, on that data. This works (meaning that R spits out numbers), but I don't know if it's any good.

    For example, reading this article on how to perform three-level meta-analyses with the help of Metafor, it clearly states that

    [i]t is important to note that the models used above assume that the sampling errors of the effect size estimates are independent. This is typically an appropriate assumption as long as there is no overlap in the data/individuals used to compute the various estimates. However, when multiple estimates are obtained from the same group of individuals, then this assumption is most certainly violated. Similarly, when multiple treatment groups are contrasted against a common control group, then the reuse of the control group data automatically induces a correlation between the estimates. See Berkey et al. (1998) and Gleser & Olkin (2009) for examples involving correlated sampling errors.

    Looking at the suggested articles, it pretty soon becomes clear that the methods recommended involves having the variance-covariance matrix that specifies the correlations between the different effect sizes, something that I don't have and won't be able to get (or indeed even estimate).

How should I proceed?

How should/could I proceed from here? I've been reading several good articles on the topic of dependencies in meta-analyses (e.g. Konstantopoulos 2011, Cheung 2014, and Scammacca 2014) but I still find it a bit troublesome to turn all that general discussion into a concrete analysis for my specific data set. Any help would be highly appreciated.

  • $\begingroup$ I may not have followed this correctly but if you collapse over testing occasions I think you are still allowing for the correlation by having the random intercepts. $\endgroup$
    – mdewey
    Commented May 20, 2016 at 14:09
  • 1
    $\begingroup$ If you have a large dataset, then you want to use rma.mv(..., sparse=TRUE). It may still take a bit of time, but it should work. And use rma.mv(..., sparse=TRUE, verbose=TRUE) so you can see how it is progressing (it's a bit like watching grass grow). $\endgroup$
    – Wolfgang
    Commented May 20, 2016 at 14:30
  • $\begingroup$ @Wolfgang Thanks, that worked! Now I just have to resolve the dependency issues outlined in analysis number 2. I've found a thread where you come with some suggestions that I'm going to check out: stats.stackexchange.com/questions/166964/… $\endgroup$
    – Speldosa
    Commented May 23, 2016 at 10:08
  • $\begingroup$ ©mdewey Sorry, I don't follow. Could you elaborate a bit? $\endgroup$
    – Speldosa
    Commented May 23, 2016 at 10:08


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.