# Meta-Analyzing Dependent Effect Sizes (Log Response Ratios)--Is Averaging Effects An Acceptable Solution?

Given k studies, for each computing a log response ratio $L$ and variance $v$ following the Hegdes et al., 1999 equations:

$$L = ln(R) = ln(\bar{X}_{E})- ln(\bar{X}_{C})$$ $$v = \frac{(SD_{E})^2}{n_{E}\bar{X}_{E}^2}+\frac{(SD_{C})^2}{n_{C}\bar{X}_{C}^2}$$ one obtains a vector of log response ratios $L_1 ... L_k$ and corresponding variances $v_1 ... v_k$.

There is the need to aggregate some studies as they are considered to belong together as one main study. Say, studies 1, 2 and 3, each having currently an $L$ and a $v$ need to be considered actually as one study with one $L$ and one $v$.

A) How to proceed in such cases?

B) Is it ok to simply average $L_1, L_2, L_3$ and $v_1, v_2, v_3$ and get $\bar{L}_{123}$ and $\bar{v}_{123}$ that can be further used with other unaggregated $L$-s and $v$-s in the meta-analysis?

Your situation is a case of attempting to meta-analyze dependent effect sizes. I'll outline the various ways one could proceed in answering your first question, and then describe my sense of how one ought to proceed in answering your second question.

How You Could Handle Dependent Effect Sizes in Meta-Analysis

Cheung (2014) has a nice summary of the simpler ways in which researchers have attempted to handle dependent effect sizes in their meta-analyses. They include:

1. Ignoring Dependence: pretend all the effect sizes are independent, and conduct your meta-analysis accordingly
2. Average Dependent Effect Sizes: this is the method you are inquiring about, and there are several approaches to conducting meta-analyses in this way.
3. Choose one effect size: if you don't want to ignore dependence, or average across it, you can somehow choose one effect size of your dependent bunch to serve as the effect included in your meta-analysis
4. Shift the "unit of analysis": proposed by Cooper (2010), this essentially is strategic averaging, whereby you would aggregate across effect sizes (e.g., produced by studies from the same lab) when attempting to estimate an effect at a higher unit of analysis (e.g., the overall effect across labs).

There are also three more sophisticated means of handling dependent effect sizes in meta-analysis:

1. Multivariate Meta-Analysis: the most demanding method, in terms of level of information required, is the multivariate meta-analysis approach described by Kalaian & Raudenbush (1996). If you know the correlations among dependent effect sizes, then their dependency can be modelled explicitly, while facilitating every effect's inclusion in the synthesis.

2. Multilevel Meta-Analysis: the approach advocated by Cheung (2014) and others before him (e.g., Konstantopoulos, 2011). This approach partitions effect-size heterogeneity into multiple levels (e.g., between-sample heterogeneity, and within-sample heterogeneity), and does not require the correlations among dependent effect sizes to be known.

3. Robust Variance Estimation: the approach developed by Hedges et al. (2010). You include each effect size in the synthesis, and account for dependency by calculating a sampling variance of the overall effect that is "robust", by using the residuals within each sample to approximate the correlation between dependent effects.

How You Should Handle Dependent Effect Sizes in Meta-Analysis

Of the simpler methods, it's probably obvious to most that ignoring dependency among effect sizes, while using statistical methods that assume independence, is probably not a good idea.

Averaging and shifting the "unit of analysis" are also not great, because they are prone to increasing both Type I and Type II errors, limit your options for moderator testing (less so for shifting the "unit of analysis"), and may bias (underestimate) your estimation of heterogeneity (Cheung, 2014; Moeyaert et al., 2017).

Choosing one effect size might seem like a bad idea at first, but there may be cases where it's a reasonable thing to do. A colleague of mine, for example, is a clinician in an area where there are several assessment tools for the condition she specializes in treating, one of which is considered to be a field-wide "gold-standard" measure. If she were conducting a meta-analysis in this area, she could make the case that the synthesis should only include effects involving that "gold-standard" assessment, which would dictate which effect size she chose from studies reporting multiple.

Of the more sophisticated strategies, the main consideration is what information do you have on the level of dependency among effects in your synthesis. If you have all of these correlations among dependent effects, you could opt for multivariate meta-analysis. In practice, however, these correlations among dependent effects are rarely reported (Becker, 2000) and therefore multivariate meta-analyses are not terribly common (Ahn et al., 2012).

If you don't have the correlations among dependent effect sizes, that leaves you with robust variance estimation methods or multilevel meta-analysis methods. There are a couple of simulation studies comparing these approaches (López‐López et al., 2017; Moeyaert et al., 2017), including against the strategy of averaging over dependent effect sizes. The main take-aways from these simulation studies are that both methods are generally superior to averaging, and selecting between the two comes down to what statistical tests are crucial for your investigation, and how many studies you have.

Take-Away

If you are primarily interested in the overall effect, and less interested in moderators, robust variance estimation methods give you greater power with fewer studies. If you are more interested in moderators (and especially modelling heterogeneity and moderators at multiple levels of analysis), multilevel meta-analyses will give you more power to detect those moderators with fewer studies, but it is less powerful for detecting overall effects in smaller samples relative to robust variance methods. If you have a large number of studies (e.g. 60+), the differences between the two methods are greatly attenuated, so just pick the one that most directly maps onto your questions about moderators (they can both be implemented relatively easily in R).

References

Ahn, S., Ames, A. J., & Meyers, N. D. (2012). A review of meta-analysis in education: Methodological strengths and weaknesses. Review of Educational Research, 82, 436–476.

Becker, B. J. (2000). Multivariate meta-analysis. In H. E. A. Tinsley & E. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematcial modeling (pp. 499–525). Orlando, FL: Academic Press.

Cheung, M. W. L. (2014). Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychological Methods, 19(2), 211-229.

Cooper, H. M. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Los Angeles: Sage Publications, Inc.

Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta‐regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39-65.

Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed linear model for meta-analysis. Psychological Methods, 1(3), 227-235.

Konstantopoulos, S. (2011). Fixed effects and variance components estimation in three‐level meta‐analysis. Research Synthesis Methods, 2(1), 61-76.

López‐López, J. A., Van den Noortgate, W., Tanner‐Smith, E. E., Wilson, S. J., & Lipsey, M. W. (2017). Assessing meta‐regression methods for examining moderator relationships with dependent effect sizes: A Monte Carlo simulation. Research Synthesis Methods, 8(4), 435-450.

Moeyaert, M., Ugille, M., Natasha Beretvas, S., Ferron, J., Bunuan, R., & Van den Noortgate, W. (2017). Methods for dealing with multiple outcomes in meta-analysis: a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. International Journal of Social Research Methodology, 20(6), 559-572.

• Just trying to understand: why do you conclude that is only the case of dependent effect sizes? If that is the case, then don't I have to use a different formula for the variance of the log response ratio, like discussed here by @Wolfgang? Feb 10, 2018 at 11:44
• "There is the need to aggregate some studies as they are considered to belong together as one main study. Say, studies 1, 2 and ... need to be considered actually as one study" is what tips me off that there is dependency. And re: the link in your comment, that seems to be about adjusting variances in the presence of effects from both matched and randomized designs, which seems to me a different matter. Feb 10, 2018 at 16:43