I'm conducting a meta-analysis on standardised mean difference scores. Some studies provide multiple effect sizes, thereby violating the assumption of independence. An example is given below (all effect sizes were calculated with regard to a pre-test). In study A, all participants received the same treatment (watching a video), and were tested repeatedly. In study B, there were two different treatment groups (one group watched a video, the other group listened to an audio book), and everyone was tested once. Study C provided only one effect size.

study        treatment          testing_moment         effect_size

A            video              immediately            0.6
A            video              delayed                0.5
B            video              immediately            0.9
B            audio_book         immediately            0.7
C            audio_book         delayed                0.4

I'm using the metafor package in R, in which you can fit a multilevel model to account for non-independent sampling errors.

What I've done:

rma.mv(effect_size_vector, variance_vector, mods = ~ testing_moment, 
  random = ~ 1 | treatment/study, data = rev)

Could anyone please have a look whether this approach is correct? I'm especially unsure about whether I've correctly indicated the clustering using slash (/) (this decision was based on this page), and whether the model as a result indeed takes into account the non-independence of effect sizes.

I'm also wondering whether somehow it should be corrected that the samples in study A are dependent and in study B they are independent. Or is that already accounted for by virtue of the treatment variable being the same for both samples in study A?

| cite | improve this question | | | | |
  • 2
    $\begingroup$ Have you had a look at metafor-project.org/doku.php/analyses:konstantopoulos2011 which gives an example of a three level model? I think you may have the terms reversed in your random specification (your current version reads as study within treatment). Depending on your scientific question do you not want to look at treatment as a fixed effect too? $\endgroup$ – mdewey Jul 19 '16 at 11:42
  • 1
    $\begingroup$ Thank you for the help! Indeed the terms were reversed, I will change it in my analysis. Treatment is also a factor for analysis, but I simplified the question in the above example. I will look into the documentation you recommend and then get back to this post. $\endgroup$ – Johanna Jul 19 '16 at 17:19
  • 2
    $\begingroup$ If treatment only has two levels I wonder about the desirability of including it as a random effect at all. $\endgroup$ – mdewey Jul 19 '16 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.