Calculating effect size (lnR) variances for studies with different study designs for use in meta-analysis

Question

My question concerns the calculation of effect size variance for studies that are going to be included in a meta-analysis. I want to calculate variance so that I can weight each study by the inverse of it's variance in a meta-regression.

It is established that you need to calculate variance in different ways when you have different study designs. For example, it must be calculated differently when you have completely randomised and matched study designs. (Matched means systematically paired or randomised block designs, for example). In a completely randomised design, you will use sample sizes, standard deviations and means of each treatment group to calculate the variance of the effect size. In matched designs, however, you are interested in the differences between each matched pair, and the number of pairs.

Borenstein et al. (2009) tell you how to calculate variance differently for Hedge's d and g for both randomised and matched designs, but does not tell you how to calculate variance for matched designs for log response ratio effect size.

I am doing a meta-analysis investigating differences in the number of species found in control and treatment types of forest. I am using the log response ratio (lnR) to represent proportional differences in the number of species between control and treatment forests. I have studies with both randomised and matched study designs.

So my question: How do you calculate the effect size variance for the log response ratio, for studies with randomised and matched study designs. What are the different calculations?

Answer 1

Let's cover each case in turn.

Two Independent Samples

Let $\bar{x}_1$ and $\bar{x}_2$ denote the observed means in the first and second group, respectively, $s_1$ and $s_2$ the standard deviations, and $n_1$ and $n_2$ the sample sizes. Then the log-transformed ratio of means (also called log response ratio) is given by $$y = \ln(\bar{x}_1 / \bar{x}_2),$$ for which we can estimate the sampling variance with the equation $$Var[y] = \frac{s_1^2}{n_1 \bar{x}_1^2} + \frac{s_2^2}{n_2 \bar{x}_2^2}.$$ See, for examples, Hedges et al. (1999).

Two Dependent Samples

If you have two dependent samples (e.g., because the same units of analysis have been measured twice, such as before and after a particular treatment), then let $\bar{x}_1$ and $\bar{x}_2$ denote the means at the first and second measurement occasion, $s_1$ and $s_2$ analogously for the standard deviations, and now there is only $n$ for the size of the group. Again, we can define the log response ratio as $$y = \ln(\bar{x}_1 / \bar{x}_2).$$ The sampling variance can now be estimated with $$Var[y] = \frac{s_1^2}{n \bar{x}_1^2} + \frac{s_2^2}{n \bar{x}_2^2} - \frac{2 r s_1 s_2}{\bar{x}_1 \bar{x}_2 n},$$ where $r$ is the correlation of the measurements between the two measurement occasions. See Lajeunesse (2011). The same equation can be used in a matched-pairs design, except that subscripts 1 and 2 represent the two groups.

Note that you will need an estimate of the correlation to use this equation. If it is not reported or can be derived based on other information reported in a study, you could try contacting the authors. Alternatively, you may just have to make a reasonable guess and then conduct a sensitivity analysis in the end to make sure that the conclusions from the meta-analysis do not depend on the guess.

References

Hedges, L. V., Gurevitch, J., & Curtis, P. S. (1999). The meta-analysis of response ratios in experimental ecology. Ecology, 80, 1150-1156.

Lajeunesse, M. J. (2011). On the meta-analysis of response ratios for studies with correlated and multi-group designs. Ecology, 92, 2049-2055.