Completely edited: If a meta-analysis include many studies that reported multiple effect sizes within each study, multi-level meta-analysis is one way to account for this dependency in effect sizes. However, there seems to be two approaches: one that utilize the traditional multilevel models and another that utilize multilevel structural equation modeling (uses metaSEM package; Cheung, 2014)

In regular multi-level modeling, the three levels are:
Level 1- effect size
Level 2- within-study variation
Level 3- between-study variation

However, it's unclear to me what the three levels are for the multilevel SEM approach. Is it the same? Some papers that used metaSEM and cited Cheung 2014 (e.g., Lebuda et al., 2016) describes the three levels as:
Level 1- study participants
Level 2- effect sizes (within-study variation)
Level 3- study (between-study variation)

My questions are:
1. Did Lebuda et al., 2016 have it right that those are indeed the three levels?
2. If so, how is it possible to have level 1 as study participants given that in meta-analysis you typically don't have data at the participant level?
3. What's the added advantage to using multilevel SEM vs. just multilevel approach?

Weisz, J. R., Kuppens, S., Ng, M. Y., Eckshtain, D., Ugueto, A. M., Vaughn-Coaxum, R., ... & Weersing, V. R. (2017). What five decades of research tells us about the effects of youth psychological therapy: A multilevel meta-analysis and implications for science and practice. American Psychologist, 72(2), 79.

Cheung, M. W. L. (2014). Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychological Methods, 19(2), 211.

metaSEM example

Paper that used metaSEM to do multilevel SEM meta-analysis and describes the three levels:
Lebuda, I., Zabelina, D. L., & Karwowski, M. (2016). Mind full of ideas: A meta-analysis of the mindfulness–creativity link. Personality and Individual Differences, 93, 22-26.

  • $\begingroup$ Can you give us some more details of what is troubling you as from your description I am not sure whether Cheung and Weisz are actually describing the same situation. Perhaps you could give an example? Also providing references for the two papers you cite would be very helpful. $\endgroup$
    – mdewey
    Commented Jun 10, 2018 at 10:55
  • $\begingroup$ @mdewey Thank you so much for your response!/ $\endgroup$
    – phoebe
    Commented Jun 11, 2018 at 4:32
  • $\begingroup$ @mdewey Sorry, I'm new to stackexchange, so I didn't know hitting enter would automatically submit the comment. Thanks for responding! Because of characters limit, I elaborated in the original post instead of down here. Please let me know if I'm still not making sense! Any help will be very much appreciated! $\endgroup$
    – phoebe
    Commented Jun 11, 2018 at 4:47

1 Answer 1

  1. Those are indeed the three levels. When Lebuda says'study participants', he/she means that the variance at the first level refers to the sampling variance, and this variance is directly related to the number of participants of a given study (that is why it is called 'study participants' level). The sampling variance is assumed to be known and estimated in advance.
    1. I think this point is already answered with the previous. It's true that in meta-analysis we don't normally have the raw data. However, we do have the sample size, and with that sample size (and with some other data) we can calculate the sampling variance, which is the variance at first level.
    2. Both approaches are practically the same, both are based on the same regression model. The difference is that Cheung(2014) proposes to use SEM techniques, and that makes his approach more flexible in the sense that constrains among parameters can be imposed. Cheung talks more about it in this article:

Cheung, M. W.-L. (2014). Modeling dependent effect sizes with threelevel meta-analyses: A structural equation modeling approach. Psychological Methods, 19, 211–229. https://doi.org/10.1037/ a0032968

Summarizing: unless you want to fit a model with constrains among some parameters, both methods work equally good.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.