# Calculating effect sizes and standard errors for the difference between two standardized mean differences

I have two related questions, both of which are related to a meta-analysis I am performing where where the primary outcomes are expressed in terms of the standardized mean difference.

My studies have multiple variables that are available with which to calculate the standardized mean difference. I would like to know the extent to which the standardized mean differences calculated on one variable are consistent with the standardized mean differences on the other. To my mind, this question could be expressed as as a meta-analysis on the difference between two sets of standardized mean differences. However, I am having trouble determining the effect size and sampling error for the difference between two standardized mean differences within the same study.

To express my problem in a different way, consider a two-condition study with groups $g_1$ and $g_2$ and outcome variables $var_1$ and $var_2$. These two outcome variables are correlated as $cor(var_1, var_2)$. We can calculate standardized mean differences for $var_1$ and $var_2$ across $g_1$ and $g_2$, yielding $d_{var1}$, $d_{var_2}$, and their sampling variances $v_{d_{var_1}}$ and $v_{d_{var_2}}$. I have included a very simple schematic of the situation below.

Now let's say that we calculate a difference between $var_1$ and $var_2$ as $diff$. I can calculate the standardized mean difference across $g_1$ and $g_2$ as $d_{diff}$, which has sampling variance $v_{d_{diff}}$.

What I would like to do is express $d_{diff}$ and $v_{d_{diff}}$ in terms of the following variables:

1. Effect sizes $d_{var_1}$ and $d_{var_2}$,
2. Sampling variances $v_{d_{var_1}}$ and $v_{d_{var_2}}$, and
3. Correlation $cor(var_1, var_2)$

I feel like this goal should be possible given the fact that, in a simple (non meta-analytic) context, the standard deviation of the difference between $var_1$ and $var_2$ is given as

$sd(var_1)^2 + sd(var_2)^2 - 2 * cor(var_1, var_2) * sd(var_1) * sd(var_2)$

I am also interested in a slightly more complicated situation where one has studies with 3 (or more) groups, and where one therefore calculates two sets of standardized mean differences between the two candidate variables.

To express this second question in a different way, assume that a given study has three groups $g_1$, $g_2$, and $g_3$ and two outcome variables $var_1$ and $var_2$. Furthermore, assume once again that $var_1$ and $var_2$ are correlated as $cor(var_1, var_2)$.

Choose group $g_1$ as the reference group and, for $var_1$, calculate effect sizes for group $g_1$ vs $g_2$ and $g_1$ vs $g_3$. This will yield two sets of effect sizes for each of $var_1$ and $var_2$ -- for $var_1$, $d_{var1_{g_1 - g_2}}$ and $d_{var1_{g_1 - g_3}}$, and, for $var_2$, $d_{var2_{g_1 - g_2}}$ and $d_{var2_{g_1 - g_3}}$. This will also yield two sampling variances for each set of effect sizes (for $var_1$, $v_{d_{var1_{g_1 - g_2}}}$ and $v_{d_{var1_{g_1 - g_3}}}$, and, for $var_2$, $v_{d_{var2_{g_1 - g_2}}}$ and $v_{d_{var2_{g_1 - g_3}}}$) and one sampling covariance for each variable (for $var_1$, $cov(d_{var1_{g_1 - g_2}}, d_{var1_{g_1 - g_3}})$, and, for $var_2$, $cov(d_{var2_{g_1 - g_2}}, d_{var2_{g_1 - g_3}})$). I have included a very simple schematic of the situation below.

Once again, I can create a difference score between $var_1$ and $var_2$, yielding $diff$. I can then calculate two sets of effect sizes on this difference score as above, calculating a standardized mean difference for the comparison between $g_1$ and $g_2$ (yielding $d_{diff_{g_1 - g_2}}$) and a standardized mean difference for the comparison between $g_1$ and $g_3$ (yielding $d_{diff_{g_1 - g_3}})$. This procedure will, of course, also yield corresponding sampling variances and covariances.

What I would like is to express the effect sizes, sampling variances, and sampling covariances for $diff$ in terms of:

1. Effect sizes $d_{var1_{g_1 - g_2}}$, $d_{var1_{g_1 - g_3}}$, $d_{var2_{g_1 - g_2}}$, and $d_{var2_{g_1 - g_3}}$
2. Sampling variances $v_{d_{var1_{g_1 - g_2}}}$, $v_{d_{var1_{g_1 - g_3}}}$, $v_{d_{var2_{g_1 - g_2}}}$, and $v_{d_{var2_{g_1 - g_3}}}$,
3. Sampling covariances $cov(d_{var1_{g_1 - g_2}}, d_{var1_{g_1 - g_3}})$ and $cov(d_{var2_{g_1 - g_2}}, d_{var2_{g_1 - g_3}})$, and
4. Correlation $cor(var_1, var_2)$

Once again, I feel like my goal should be feasible given the fact that it is possible to calculate the standard deviation of a difference score between $var_1$ and $var_2$ given $sd(var_1)$, $sd(var_2)$, and $cor(var_1, var_2)$.

I realize that my questions are a little complicated, but I feel like they could be answered given a bit of clever algebra. Let me know if I can clarify my question and / or notation in any way.

I can certainly give you an answer to the first part of your question.

Using your notation, let $d_{var_1}$ and $d_{var_2}$ denote the two d values (computed for the same two groups) based on two different dependent variables and let $v_{var_1}$ and $v_{var_2}$ denote the corresponding sampling variances, which are typically computed/estimated with: $$v_{var_1} = \frac{1}{n_1} + \frac{1}{n_2} + \frac{d_{var_1}^2}{2(n_1 + n_2)}$$ and $$v_{var_2} = \frac{1}{n_1} + \frac{1}{n_2} + \frac{d_{var_2}^2}{2(n_1 + n_2)},$$ where $n_1$ and $n_2$ are the two group sizes.

Let $r = cor(var_1, var_2)$ denote the correlation between the two variables. Then the covariance between the two d values can be estimated with: $$cov(d_{var_1}, d_{var_2}) = \left(\frac{1}{n_1} + \frac{1}{n_2}\right)r + \left(\frac{d_{var_1}d_{var_2}}{2(n_1 + n_2)} \right)r^2.$$ See equation (19.27) in the chapter on stochastically dependent effect sizes by Gleser and Olkin (2009) in The handbook of research synthesis and meta-analysis (2nd ed.).

Therefore, the sampling variance of $$d_{diff} = d_{var_1} - d_{var_2}$$ can be computed/estimated with: $$v_{d_{diff}} = v_{var_1} + v_{var_2} - 2 cov(d_{var_1}, d_{var_2}).$$

The chapter by Gleser and Olkin also partly addresses your second question. Essentially, you have what the authors call a 'multiple-treatment study' and they provide equations for the covariance in that case as well (see Expectation of correlated variables). However, your case is actually a combination of the 'multiple-treatment' and 'multiple-endpoint' case. Deriving the necessary covariance equations would require some additional work.

• Is it also the case that $d_{diff}$ is simply $d_{var_1}$ - $d_{var_2}$, or is this quantity also affected by $cor(var_1, var_2)$? – Patrick S. Forscher Jul 17 '14 at 21:18
• Yes, just take the difference. The correlation is only relevant for computing the sampling variance of $d_{diff}$. – Wolfgang Jul 17 '14 at 21:59
• Great! Do you, by chance, have any idea of what approach I might take to resolve my second question? I know from Gleser and Olkin that the covariance between $d_{g_1 - g_2}$ and $d_{g_1 - g_3}$ is $1/n_{g_1} + (d_{g_1 - g_2} * d_{g_1 - g_3}) / (2 * (n_{g_1} + n_{g_2} + n_{g_3}))$, but it's not clear to me how to use this information to obtain the covariance between the two effect sizes for $d_{diff}$. – Patrick S. Forscher Jul 17 '14 at 22:24
• If you don't know the full solution, I would be happy to have a general idea of how I would investigate this issue. – Patrick S. Forscher Jul 17 '14 at 22:26
• You would have to go back to the derivation of those covariances and see if you can generalize/combine the two cases (multiple groups and multiple endpoints). A brief sketch on how the covariance for the multiple endpoint case can be derived can be found in the appendix of Rosenthal and Rubin (1986). I don't know of a reference that covers the multiple group case. – Wolfgang Jul 18 '14 at 10:19

This question can be answered by using a structural equation modeling (SEM) approach. It can be used as long as the effect sizes are functions of the parameters, such as means, correlations, and standard deviations. The sampling covariance matrix is numerically derived by the use of Delta method automatically in SEM. Chapter 3 of Cheung (2015) provides an introduction and examples in this approach.

One of the examples used in the book is the multiple treatment multiple-endpoint studies. Here are the syntax and output in R.

###################################################
### code chunk number 8: ME_MT
###################################################

## Load the library for SEM
library(lavaan)

## Covariance matrix of the control group for variables 1 and 2
lower <- '11
5, 10'
## Convert a lower triangle data into a covariance matrix
Cov1 <- getCov(lower, diag=TRUE, names=c("x1", "x2"))

## Covariance matrix of the treatment group 1 for variables 1 and 2
lower <- '12
6, 11'
Cov2 <- getCov(lower, diag=TRUE, names=c("x1", "x2"))

## Covariance matrix of the treatment group 2 for variables 1 and 2
lower <- '13
7, 12'
Cov3 <- getCov(lower, diag=TRUE, names=c("x1", "x2"))

## Convert covariance matrices into a list
Cov <- list(Cov1, Cov2, Cov3)

## Means for the three groups
## 10 and 11 are the means for variables 1 and 2
Mean <- list(c(10,11), c(12,13), c(13,14))

## Sample sizes for the groups
N <- c(50, 50, 50)

## Assuming homogeneity of covariance matrices
## You can free this constraint by using different labels
model5 <- 'eta1 =~ c("sd1", "sd1", "sd1")*x1
eta2 =~ c("sd2", "sd2", "sd2")*x2
eta1 ~~ c("r", "r", "r")*eta2
## The subscripts 0, 1 and 2 represent the means
##  of the control and two  treatment groups
x1 ~ c("m1_0", "m1_1", "m1_2")*1
x2 ~ c("m2_0", "m2_1", "m2_2")*1
## The measurement errors are fixed at 0
x1 ~~ 0*x1
x2 ~~ 0*x2
## Multiple endpoint effect size 1 for treatment group 1
ES1_1 := (m1_1 - m1_0)/sd1
## Multiple endpoint effect size 2 for treatment group 1
ES2_1 := (m2_1 - m2_0)/sd2
## Multiple endpoint effect size 1 for treatment group 2
ES1_2 := (m1_2 - m1_0)/sd1
## Multiple endpoint effect size 2 for treatment group 2
ES2_2 := (m2_2 - m2_0)/sd2'

fit5 <- sem(model5, sample.cov=Cov, sample.mean=Mean,
sample.nobs=N, std.lv=TRUE,
sample.cov.rescale=FALSE)

## Obtain the free parameters in the model
( x <- fit5@Fit@x )

## [1]  3.464102  3.316625  0.522233 10.000000 11.000000 12.000000 13.000000
## [8] 13.000000 14.000000

## Obtain the sampling covariance matrix of the parameter estimates
VCOV <- vcov(fit5)

## Compute the multivariate effect sizes
( ES <- fit5@Model@def.function(x=x) )
##     ES1_1     ES2_1     ES1_2     ES2_2
## 0.5773503 0.6030227 0.8660254 0.9045340

## Compute the jacobian for 'defined parameters'
JAC <- lavaan:::lavJacobianD(func=fit5@Model@def.function, x=x)

## Compute the sampling covariance matrix using delta method
ES.VCOV <- JAC %*% VCOV %*% t(JAC)

## Add the variable names for ease of reference
dimnames(ES.VCOV) <- list(names(ES), names(ES))

ES.VCOV
##            ES1_1      ES2_1      ES1_2      ES2_2
## ES1_1 0.04111111 0.02120582 0.02166667 0.01091942
## ES2_1 0.02120582 0.04121212 0.01091942 0.02181818
## ES1_2 0.02166667 0.01091942 0.04250000 0.02160145
## ES2_2 0.01091942 0.02181818 0.02160145 0.04272727


In this example, the estimated vector of effect sizes are their sampling covariance matrix are ES and ES.VCOV, respectively. ES1_1 and ES2_1 are the effect sizes for group 1 comparing against the control group, while ES1_2 and ES2_2 are the effect sizes group 2 comparing against the control group.

Reference

Cheung, M. W.-L. (2015). Meta-analysis: A structural equation modeling approach. Chichester, West Sussex: John Wiley & Sons, Inc..

• Thanks for sharing this interesting approach! When I tried to run the following lines: ( ES <- fit5@Model@def.function(x=x) ) and JAC <- lavaan:::lavJacobianD(func=fit5@Model@def.function, x=x) I got an error that x does not exist. – Patrick S. Forscher Jan 6 '15 at 16:37
• Also, your example seems to suggest that, for this approach to work, I need to know the correlation/covariance between var1 and var2 within g1, g2, and g3. Is this the case? Usually in the studies I'm working with, only the overall correlation (collapsing across g1, g2, and g3) is reported. – Patrick S. Forscher Jan 6 '15 at 16:40
• Finally, does this approach work in the two-group case where I do not know the means and standard deviations for $var_1$ and $var_2$, but am able to directly extract effect sizes $d_{var_1}$ and $d_{var_2}$ from, for example, t-tests reported in the paper in question? – Patrick S. Forscher Jan 6 '15 at 16:44
• Thanks, Patrick. I have added the missing line: (x <- fit5@Fit@x). Since the effect sizes are functions of means, variances, and covariances, this approach needs these elements. If some of these elements are not available, you may need to figure other approaches... – Mike Cheung Jan 8 '15 at 10:43
• Hi Mike, I hope you are still following this thread. I was interested in your approach, so I tried simulating some three-group data with two variables (code pasted in the comment below). When I compared your approach to some manual calculations, I obtained identical effect sizes but different sampling errors of the effect sizes. As far as I can tell, I'm using your code and the correct formulas for the sampling variances. Any idea what's going on? – Patrick S. Forscher Feb 3 '15 at 15:35

I am not completely certain how this solution was derived, but I thought I would post it anyway so that other people could evaluate it. I also thought that this information was worth posting as a full answer rather than leaving it buried in the comments of the answer provided by @Wolfgang.

According to a response that Ian White supplied in correspondence with me, given groups $g_1$, $g_2$, and $g_3$, and assuming that the standard deviation used to calculate one's effect sizes was pooled across $g_1$, $g_2$, and $g_3$, $$cov(d_{diff_{g_1 - g_2}}, d_{diff_{g_1 - g_3}}) = \frac{r}{n_1} + \frac{d_{diff_{g_1 - g_2}} * d_{diff_{g_1 - g_3}} * r^2}{(2 * (n_1 + n_2 + n_3))}$$

Again, I'm not entirely certain how this solution was derived, and I would be grateful for any insight that others could provide.