Calculating effect sizes and standard errors for the difference between two standardized mean differences

Question

I have two related questions, both of which are related to a meta-analysis I am performing where where the primary outcomes are expressed in terms of the standardized mean difference.

My studies have multiple variables that are available with which to calculate the standardized mean difference. I would like to know the extent to which the standardized mean differences calculated on one variable are consistent with the standardized mean differences on the other. To my mind, this question could be expressed as as a meta-analysis on the difference between two sets of standardized mean differences. However, I am having trouble determining the effect size and sampling error for the difference between two standardized mean differences within the same study.

To express my problem in a different way, consider a two-condition study with groups $g_1$ and $g_2$ and outcome variables $var_1$ and $var_2$. These two outcome variables are correlated as $cor(var_1, var_2)$. We can calculate standardized mean differences for $var_1$ and $var_2$ across $g_1$ and $g_2$, yielding $d_{var1}$, $d_{var_2}$, and their sampling variances $v_{d_{var_1}}$ and $v_{d_{var_2}}$. I have included a very simple schematic of the situation below.

Now let's say that we calculate a difference between $var_1$ and $var_2$ as $diff$. I can calculate the standardized mean difference across $g_1$ and $g_2$ as $d_{diff}$, which has sampling variance $v_{d_{diff}}$.

What I would like to do is express $d_{diff}$ and $v_{d_{diff}}$ in terms of the following variables:

1. Effect sizes $d_{var_1}$ and $d_{var_2}$,
2. Sampling variances $v_{d_{var_1}}$ and $v_{d_{var_2}}$, and
3. Correlation $cor(var_1, var_2)$

I feel like this goal should be possible given the fact that, in a simple (non meta-analytic) context, the standard deviation of the difference between $var_1$ and $var_2$ is given as

$sd(var_1)^2 + sd(var_2)^2 - 2 * cor(var_1, var_2) * sd(var_1) * sd(var_2)$

---

I am also interested in a slightly more complicated situation where one has studies with 3 (or more) groups, and where one therefore calculates two sets of standardized mean differences between the two candidate variables.

To express this second question in a different way, assume that a given study has three groups $g_1$, $g_2$, and $g_3$ and two outcome variables $var_1$ and $var_2$. Furthermore, assume once again that $var_1$ and $var_2$ are correlated as $cor(var_1, var_2)$.

Choose group $g_1$ as the reference group and, for $var_1$, calculate effect sizes for group $g_1$ vs $g_2$ and $g_1$ vs $g_3$. This will yield two sets of effect sizes for each of $var_1$ and $var_2$ -- for $var_1$, $d_{var1_{g_1 - g_2}}$ and $d_{var1_{g_1 - g_3}}$, and, for $var_2$, $d_{var2_{g_1 - g_2}}$ and $d_{var2_{g_1 - g_3}}$. This will also yield two sampling variances for each set of effect sizes (for $var_1$, $v_{d_{var1_{g_1 - g_2}}}$ and $v_{d_{var1_{g_1 - g_3}}}$, and, for $var_2$, $v_{d_{var2_{g_1 - g_2}}}$ and $v_{d_{var2_{g_1 - g_3}}}$) and one sampling covariance for each variable (for $var_1$, $cov(d_{var1_{g_1 - g_2}}, d_{var1_{g_1 - g_3}})$, and, for $var_2$, $cov(d_{var2_{g_1 - g_2}}, d_{var2_{g_1 - g_3}})$). I have included a very simple schematic of the situation below.

Once again, I can create a difference score between $var_1$ and $var_2$, yielding $diff$. I can then calculate two sets of effect sizes on this difference score as above, calculating a standardized mean difference for the comparison between $g_1$ and $g_2$ (yielding $d_{diff_{g_1 - g_2}}$) and a standardized mean difference for the comparison between $g_1$ and $g_3$ (yielding $d_{diff_{g_1 - g_3}})$. This procedure will, of course, also yield corresponding sampling variances and covariances.

What I would like is to express the effect sizes, sampling variances, and sampling covariances for $diff$ in terms of:

1. Effect sizes $d_{var1_{g_1 - g_2}}$, $d_{var1_{g_1 - g_3}}$, $d_{var2_{g_1 - g_2}}$, and $d_{var2_{g_1 - g_3}}$
2. Sampling variances $v_{d_{var1_{g_1 - g_2}}}$, $v_{d_{var1_{g_1 - g_3}}}$, $v_{d_{var2_{g_1 - g_2}}}$, and $v_{d_{var2_{g_1 - g_3}}}$,
3. Sampling covariances $cov(d_{var1_{g_1 - g_2}}, d_{var1_{g_1 - g_3}})$ and $cov(d_{var2_{g_1 - g_2}}, d_{var2_{g_1 - g_3}})$, and
4. Correlation $cor(var_1, var_2)$

Once again, I feel like my goal should be feasible given the fact that it is possible to calculate the standard deviation of a difference score between $var_1$ and $var_2$ given $sd(var_1)$, $sd(var_2)$, and $cor(var_1, var_2)$.

I realize that my questions are a little complicated, but I feel like they could be answered given a bit of clever algebra. Let me know if I can clarify my question and / or notation in any way.

Answer 1

I can certainly give you an answer to the first part of your question.

Using your notation, let $d_{var_1}$ and $d_{var_2}$ denote the two d values (computed for the same two groups) based on two different dependent variables and let $v_{var_1}$ and $v_{var_2}$ denote the corresponding sampling variances, which are typically computed/estimated with: $$v_{var_1} = \frac{1}{n_1} + \frac{1}{n_2} + \frac{d_{var_1}^2}{2(n_1 + n_2)}$$ and $$v_{var_2} = \frac{1}{n_1} + \frac{1}{n_2} + \frac{d_{var_2}^2}{2(n_1 + n_2)},$$ where $n_1$ and $n_2$ are the two group sizes.

Let $r = cor(var_1, var_2)$ denote the correlation between the two variables. Then the covariance between the two d values can be estimated with: $$cov(d_{var_1}, d_{var_2}) = \left(\frac{1}{n_1} + \frac{1}{n_2}\right)r + \left(\frac{d_{var_1}d_{var_2}}{2(n_1 + n_2)} \right)r^2.$$ See equation (19.27) in the chapter on stochastically dependent effect sizes by Gleser and Olkin (2009) in The handbook of research synthesis and meta-analysis (2nd ed.).

Therefore, the sampling variance of $$d_{diff} = d_{var_1} - d_{var_2}$$ can be computed/estimated with: $$v_{d_{diff}} = v_{var_1} + v_{var_2} - 2 cov(d_{var_1}, d_{var_2}).$$

The chapter by Gleser and Olkin also partly addresses your second question. Essentially, you have what the authors call a 'multiple-treatment study' and they provide equations for the covariance in that case as well (see Expectation of correlated variables). However, your case is actually a combination of the 'multiple-treatment' and 'multiple-endpoint' case. Deriving the necessary covariance equations would require some additional work.

Answer 2

This question can be answered by using a structural equation modeling (SEM) approach. It can be used as long as the effect sizes are functions of the parameters, such as means, correlations, and standard deviations. The sampling covariance matrix is numerically derived by the use of Delta method automatically in SEM. Chapter 3 of Cheung (2015) provides an introduction and examples in this approach.

One of the examples used in the book is the multiple treatment multiple-endpoint studies. Here are the syntax and output in R.

###################################################
### code chunk number 8: ME_MT
###################################################

## Load the library for SEM
library(lavaan)

## Covariance matrix of the control group for variables 1 and 2
lower <- '11          
          5, 10'
## Convert a lower triangle data into a covariance matrix
Cov1 <- getCov(lower, diag=TRUE, names=c("x1", "x2"))

## Covariance matrix of the treatment group 1 for variables 1 and 2
lower <- '12          
          6, 11'
Cov2 <- getCov(lower, diag=TRUE, names=c("x1", "x2"))

## Covariance matrix of the treatment group 2 for variables 1 and 2
lower <- '13          
          7, 12'
Cov3 <- getCov(lower, diag=TRUE, names=c("x1", "x2"))

## Convert covariance matrices into a list
Cov <- list(Cov1, Cov2, Cov3)

## Means for the three groups
## 10 and 11 are the means for variables 1 and 2
Mean <- list(c(10,11), c(12,13), c(13,14))

## Sample sizes for the groups
N <- c(50, 50, 50)

## Assuming homogeneity of covariance matrices
## You can free this constraint by using different labels
model5 <- 'eta1 =~ c("sd1", "sd1", "sd1")*x1
           eta2 =~ c("sd2", "sd2", "sd2")*x2
           eta1 ~~ c("r", "r", "r")*eta2
           ## The subscripts 0, 1 and 2 represent the means
           ##  of the control and two  treatment groups
           x1 ~ c("m1_0", "m1_1", "m1_2")*1
           x2 ~ c("m2_0", "m2_1", "m2_2")*1
           ## The measurement errors are fixed at 0
           x1 ~~ 0*x1
           x2 ~~ 0*x2
           ## Multiple endpoint effect size 1 for treatment group 1
           ES1_1 := (m1_1 - m1_0)/sd1
           ## Multiple endpoint effect size 2 for treatment group 1
           ES2_1 := (m2_1 - m2_0)/sd2
           ## Multiple endpoint effect size 1 for treatment group 2
           ES1_2 := (m1_2 - m1_0)/sd1
           ## Multiple endpoint effect size 2 for treatment group 2
           ES2_2 := (m2_2 - m2_0)/sd2'

fit5 <- sem(model5, sample.cov=Cov, sample.mean=Mean, 
            sample.nobs=N, std.lv=TRUE, 
            sample.cov.rescale=FALSE)

## Obtain the free parameters in the model
( x <- fit5@Fit@x )

## [1]  3.464102  3.316625  0.522233 10.000000 11.000000 12.000000 13.000000
## [8] 13.000000 14.000000    

## Obtain the sampling covariance matrix of the parameter estimates
VCOV <- vcov(fit5)

## Compute the multivariate effect sizes
( ES <- fit5@Model@def.function(x=x) )
##     ES1_1     ES2_1     ES1_2     ES2_2 
## 0.5773503 0.6030227 0.8660254 0.9045340

## Compute the jacobian for 'defined parameters'
JAC <- lavaan:::lavJacobianD(func=fit5@Model@def.function, x=x)

## Compute the sampling covariance matrix using delta method
ES.VCOV <- JAC %*% VCOV %*% t(JAC)

## Add the variable names for ease of reference
dimnames(ES.VCOV) <- list(names(ES), names(ES))

ES.VCOV
##            ES1_1      ES2_1      ES1_2      ES2_2
## ES1_1 0.04111111 0.02120582 0.02166667 0.01091942
## ES2_1 0.02120582 0.04121212 0.01091942 0.02181818
## ES1_2 0.02166667 0.01091942 0.04250000 0.02160145
## ES2_2 0.01091942 0.02181818 0.02160145 0.04272727

In this example, the estimated vector of effect sizes are their sampling covariance matrix are ES and ES.VCOV, respectively. ES1_1 and ES2_1 are the effect sizes for group 1 comparing against the control group, while ES1_2 and ES2_2 are the effect sizes group 2 comparing against the control group.

Reference

Cheung, M. W.-L. (2015). Meta-analysis: A structural equation modeling approach. Chichester, West Sussex: John Wiley & Sons, Inc..

Answer 3

I am not completely certain how this solution was derived, but I thought I would post it anyway so that other people could evaluate it. I also thought that this information was worth posting as a full answer rather than leaving it buried in the comments of the answer provided by @Wolfgang.

According to a response that Ian White supplied in correspondence with me, given groups $g_1$, $g_2$, and $g_3$, and assuming that the standard deviation used to calculate one's effect sizes was pooled across $g_1$, $g_2$, and $g_3$, $$cov(d_{diff_{g_1 - g_2}}, d_{diff_{g_1 - g_3}}) = \frac{r}{n_1} + \frac{d_{diff_{g_1 - g_2}} * d_{diff_{g_1 - g_3}} * r^2}{(2 * (n_1 + n_2 + n_3))}$$

Again, I'm not entirely certain how this solution was derived, and I would be grateful for any insight that others could provide.