I am currently conducting a meta-analysis where we have several studies that report multiple outcomes from the same participants. We've been advised to use the rma.mv function from the metafor package in R. I have two questions about this:

1) We don't know the covariances between the different outcomes in the same studies (they're not reported in most papers). Until recently, I was thinking that the dependency among these outcomes would be accounted for by specifying a random effects structure like random = ~ 1 | study. However, I've been reading that we should also be specifying the covariances between outcomes, and instead specify a random effects structure like random = ~ outcome | study. What is the difference between these two approaches? Will simply specifying a random effect for study without knowing the covariances between outcomes within a study result in inaccurate estimates? Alternately, would it be possible/a good idea to specify a random effects structure like random = ~ outcome | study without including the covariances between outcomes? Not all studies measure all outcomes in our meta-analysis.

2) I would like to calculate how much heterogeneity in effect sizes comes from between-study variance and how mcuh comes from within-study variance. Is there a statistic that rma.mv can give me to tell me this? I understand that Cochran's Q is a test of heterogeneity, but can it be partialled into between- and within-study heterogeneity?

Thank you!

  • $\begingroup$ What effect sizes are you intending to use in your meta-analysis, Holly (e.g., mean difference, standardized mean difference)? Also, what kind of data were you able to extract from each study for your outcomes (e.g., mean and standard deviations of outcome values in each treatment group)? Do all studies have two treatment groups? It's difficult to answer your question without specifics such as the ones I indicated. $\endgroup$ – Isabella Ghement Sep 25 '18 at 17:27
  • $\begingroup$ Sorry about that! We are using correlations as our effect size measure, so no, we don't have two treatment groups in each study. Instead we have at least one correlation per study (more if they measured multiple outcomes). Edit: to clarify, the studies use a wide variety of experimental or correlational designs. So we have extracted a variety of types of data (e.g., means and SDs for studies with different groups; regression coefficients or correlations from studies with correlational designs), and then converted these all into correlations. $\endgroup$ – Holly Engstrom Sep 25 '18 at 17:31
  • $\begingroup$ Instead (or on top) of metafor I would recommend you to try mvmeta for Stata or R, which can also accomodate studies with multiple outcomes despite the lack of details on covariance. $\endgroup$ – Joe_74 Sep 26 '18 at 8:54
  • $\begingroup$ You might find the answer to this question stats.stackexchange.com/questions/187197/… helpful and also see his comment under it. $\endgroup$ – mdewey Sep 26 '18 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.