4
$\begingroup$

I have a data set that I'm going to use for a meta-analysis that has a lot of dependencies between the effect sizes. Because of this, I've started to look into how people normally handle this, resulting in me finding a couple of articles (e.g. Konstantopoulos 2010; Cheung 2014) about three-level meta-analyses.

However, what puzzles me with all of these articles is that they're always talking about three-level meta-analyses rather than starting from the more general n-level meta-analyses. For my data set, I have what probably would count as a four-level structure (effect sizes clustered in testing occasions; testing occasions clustered into samples of participants; and samples of participants clustered into published papers) but I never find any examples of this in the literature. As of now, I'm shifting the unit of analysis by creating grand means on the sample level (meaning that I have several effect sizes for the same sample, corresponding to different factors of the certain moderator I'm interested in for the time being), and then I apply some kind of three-level analysis on that data, but I've never seen any examples of this in the literature either.

Am I missing something here?

References

  • Cheung (2014). Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychological Methods, 2, 211-229.
  • Konstantopoulos (2011). Fixed effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods, 2, 61-76.
$\endgroup$
  • 4
    $\begingroup$ You can fit multilevel models with as many levels as you deem suitable for your data with rma.mv() in the metafor package in R. $\endgroup$ – Wolfgang May 18 '16 at 17:58
3
$\begingroup$

Since the multi-level meta-analytic models are special cases of the more general multi-level models then it must be possible to fit them. People have only recently started to publish articles about multi-level models in meta-analysis so I suspect the answer to your question lies in the fact that to need such a model and to be able to fit it demands a large number of primary studies and in many fields of scientific enquiry this is rare.

$\endgroup$
  • 1
    $\begingroup$ It may not be about the statistical theory. Substantive applications are usually lack behind methodological development. I have recently collaborated in a meta-analysis using a three-level meta-analysis where the three-level meta-analysis is clearly required in the data. The paper was declined for review by an editor because the paper was considered as "methodological" rather than "substantive." $\endgroup$ – Mike Cheung May 19 '16 at 4:45
  • $\begingroup$ @MikeCheung and if you submit it to a statistical journal they will say it is not sufficiently statistical. The woes of being ahead of the curve. $\endgroup$ – mdewey May 19 '16 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.