4
$\begingroup$

I'm looking for a way to combine effect sizes (mean difference) and variances for several single studies but I don't want to conduct an overall meta-analysis. That means I'm only looking for an average score for each study.

Example:
Study 1 has 2 outcomes -> average ES and variance of ES for Study 1
Study 2 has 4 outcomes -> average ES and variance of ES for Study 2

Problem: I can calculate the mean of the effect sizes but that does not work for the variance. To average the variances I would need the correlations between the outcomes but these are rarely reported.
Also please note that the outcomes within each study are very similar (e.g. two different measures for word problem solving).

Ideas so far (I also refer to this post):

  • Estimate the covariance between outcomes using the formula by Gleser, L. J., & Olkin, I. (2009) (example in metafor): In principle this would work. However in some of my studies the sample sizes are not the same for every outcome. So I can't really apply the formula to every pair of outcome to calculate their covariance because of different Ns.
    In R-package MAd the mean of sample sizes between two outcomes is used to avoid these kind of problems. And also I have to guess the correlation between outcomes.
  • Robust Variance Estimation: Does not seem suitable to combine multiple effect sizes of only one study because the variance between studies is needed to estimate an average variance (?). And the number of studies is too small for a decent estimation.
  • Using Borenstein's (2009) formula to combine ES and variances. This would work because only effect sizes and variances of effect sizes are needed. However I would have to guess a correlation between every ES.
  • I can also do some non-statistical methods like selecting only one ES per study.

I would prefer Borenstein's method. Compared to Gleser & Olkin I also have to guess a correlation but don't need to bother with different sample sizes.
What do you think? Looking forward to your opinion!

$\endgroup$
6
$\begingroup$

The agg function in MAd package actually can apply Borenstein et al. 2009 method for you. You just need to do:

library(MAd)
agg (id = id,es = es,var = var, cor=1,method = "BHHR", data = yourdata)

With id as the column that contains the common name of all the rows you want to aggregate, and r as correlation. Depending on the type of study you are carrying out, you initial r would be different. Some people use r=0.5 as default. In my field (life sciences) I use r=1 when the non-independent outcomes are measurements across several time-points. What I would do is run the analysis with r=0, r=0.5 and r=1 and see if the conclusions are different. This is what some people would call a sensitivity analysis.

$\endgroup$
  • $\begingroup$ I stumble across the MAd package today and it saves quite some time. So you would prefer Borenstein's solution too? I also thought about doing some sensitivity analysis although with only one study estimations may vary a lot. In the end I would go for a conservative approach with a high correlation between 0.8 and 1. $\endgroup$ – Stefan Oct 29 '15 at 23:11
  • $\begingroup$ Indeed. I haven’t used Gleser & Olkin myself to be honest, whereas Borenstein et al. 2009 is frequently used in mi field and the method consistent. I would recommend Borenstein et al. (2009): Introduction to Meta Analysis. Chapter 24 Multiple Outcomes or Time-Points within a Study $\endgroup$ – fede_luppi Oct 30 '15 at 7:03
  • $\begingroup$ +1 for the sensitivity analysis. This is a good way to check if the conclusion are robust with the assumption made (r=1). $\endgroup$ – Emilie Oct 30 '15 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.