On a data set I inherited, I calculated the standardized difference between the means of 2 groups, d, for each of 5 variables. All the data was collected from same set of about 175 people, with 35 in the smaller group. What test might I use to compare the d's? One of the 5 d's is noticeably different from the others, but it could be due to chance. Thanks.

  • $\begingroup$ To clarify, I have 5 effect sizes. How do I determine if they are different to a statistically significant extent? $\endgroup$ – Joel W. Dec 14 '14 at 11:35
  • $\begingroup$ So for each of the 5 variables what is the model you assume ? Is it a an ANOVA with two groups ? With homogeneous variances or not? Are the 5 models independent of each other ? $\endgroup$ – Stéphane Laurent Dec 14 '14 at 13:27

This can be easily done with methods described in chapter 19 by Gleser and Olkin in the Handbook of Research Synthesis and Meta-Analysis (2009).

First, put the 5 d-values into a vector, $\vec{d}$.

Next, we need to construct the $5 \times 5$ variance-covariance matrix for $\vec{d}$. The diagonal elements (the variances) are given by $$Var[d_i] = \frac{1}{n_1} + \frac{1}{n_2} + \frac{d_i^2}{2(n_1 + n_2)},$$ where $n_1$ and $n_2$ are the group sizes (see equation 19.26 in the chapter). The off-diagonal elements (the covariances) are given by $$Cov[d_i, d_j] = \left(\frac{1}{n_1} + \frac{1}{n_2}\right) r_{ij} + \left(\frac{d_i d_j}{2(n_1 + n_2)}\right) r_{ij}^2,$$ where $r_{ij}$ is the correlation between variable $i$ and variable $j$ in your data (see equation 19.27 in the chapter). Let's call the resulting matrix $V$.

Now you can do several different things, including:

  1. Test the null hypothesis that the true standardized mean differences underlying these 5 observed effects are the same (i.e., $H_0: \delta_1 = \delta_2 = \ldots = \delta_5$). This is commonly called a 'test for heterogeneity' in the meta-analytic literature and can be done with the so-called $Q$-test (see equation 19.31).
  2. Test whether the true standardized mean difference for one particular effect is different from the rest (which are assumed to share the same common true effect). For this, you can fit a model that includes a dummy variable for that one effect believed to be significantly different and then test that dummy (see section 19.4.2 for regression models with such data).

I'll illustrate all of this with some made-up data and R code.

### group sizes
n1 <- 35
n2 <- 175-35

### vector with the observed d values
d <- c(.24, .10, .38, .86, .29)

### construct the var-cov matrix (R is the correlation matrix of the 5 variables)
R <- matrix(c( 1, .52, .35, .68, .44,
              NA,   1, .48, .27, .33,
              NA,  NA,   1, .56, .25,
              NA,  NA,  NA,   1, .49,
              NA,  NA,  NA,  NA,   1), nrow=5)
R[upper.tri(R)] <- t(R)[upper.tri(R)]
V <- (1/n1 + 1/n2) * R + (outer(d, d, '*') / (2*(n1 + n2))) * R^2

### load metafor package

### fit model assuming homogeneous effects
res <- rma.mv(d, V)

### examine results (esp. Q-test for heterogeneity)

### test if the 4th effect is significantly different from the rest
### note: I(1:5 == 4) gives me a dummy variable that is equal to FALSE (0) 
### for effects 1, 2, 3, and 5, and equal to TRUE (1) for effect 4
res <- rma.mv(d ~ I(1:5 == 4), V)

### examine results (esp. the p-value for the dummy variable)

So, in these data, we would reject the null hypothesis that the true effects are homogeneous ($Q(4) = 20.97$, $p = .0003$) and we would conclude that the 4th effect is significantly larger than the rest ($p < .0001$). In fact, the test for residual heterogeneity is not significant ($Q(3) = 2.23$, $p = .53$), which implies in this case that there is no significant amount of heterogeneity among effects 1, 2, 3, and 5.

One word of caution: You are picking out that one effect to test that appears to be different from the rest. But you did not choose it a priori -- you picked it after examining the effects. So, a better approach would be to consider all 5 possible tests you could have run and apply a correction for multiple testing. In the example above, the test easily survives a Bonferroni correction (just multiply the p-value for the dummy by 5), but that may not be the case in other data.

Addition: Here is code that allows you to fit all 5 models, extract the p-values, and then apply some correction for multiple testing, such as Holm's method.

pvals <- rep(NA, 5)
for (i in 1:5) {
   res <- rma.mv(d ~ I(1:5 == i), V)
   pvals[i] <- res$pval[2]
round(p.adjust(pvals, method="holm"), 4)

The results are:

[1] 0.0305 0.6274 0.3228 0.0001 0.3835

So, for these data, there is some evidence that the first and fourth values could be considered to be significantly different from the rest.

  • $\begingroup$ In short, you can reconstruct what the MANOVA would be doing, and then test various contrasts based on the MANOVA variance-covariance matrix. Is that a valid summary? $\endgroup$ – Andrew M Dec 16 '14 at 1:48
  • $\begingroup$ Probably worth noting here: given a difference in effect sizes, $\delta_1 - \delta_2 = \Delta$, if $\text{p-value}(\delta_1) \leq \alpha$ and $\text{p-value}(\delta_2) > \alpha$, it could be the case that $\text{p-value}(\Delta) \leq alpha$. $\endgroup$ – shadowtalker Dec 16 '14 at 16:32
  • $\begingroup$ @AndrewM Not quite. MANOVA tests for mean differences. The OP is asking for a way of testing the standardized mean differences against each other. That is not quite the same thing. Aside from that, yes, the approach is a sort of multivariate analysis of these effect sizes. $\endgroup$ – Wolfgang Dec 16 '14 at 18:51

I don't know any statistical test to check the difference of effect sizes. But nevertheless there is a possible solution to the problem: confidence intervals.

You can calculate the confidence interval for each of your d. If the confidence intervals have some large intersection it is OK to claim that they are similar. If they have no intersection, they are definitely not similar.

Calculating confidence intervals for effect sizes is not simple - it requires non-centrality parameters see for example wikipedia page. I use the R package MBESS

  • $\begingroup$ If there a way to compare even 2 measures of effect size? That would be a start, but one omnibus test would be better. $\endgroup$ – Joel W. Dec 14 '14 at 16:17
  • $\begingroup$ @JoelW. This is easy with a Bayesian approach but I can't help if you don't answer my questions. $\endgroup$ – Stéphane Laurent Dec 15 '14 at 10:59
  • $\begingroup$ My solution with confidence interval is really to compare 2 effect sizes. Comparing the 5 of them would require some correction for multiple correction - but that correction works for hypothesis testing - I do not know how to use it in the confidence interval approach. $\endgroup$ – Jacques Wainer Dec 16 '14 at 0:20
  • $\begingroup$ Comparing CIs is not taking the correlation between the $d$ values into consideration. $\endgroup$ – Wolfgang Dec 16 '14 at 19:00

I'm a little confused at your terminology (in particular, I think you mean d is an index for the variables, but I'm not 100% sure. You also don't provide a lot of context for how the d's are related. I'm going to assume the variables are continuous because you standardized them and compare means. Without making any other assumptions, you could try Multivariate Analysis of Variance, followed by ANOVA or t-tests for each variable assuming the MANOVA is significant.

If your data are repeated observations, MANOVA will work but you could do a longitudinal model, where you nest observations within the people. This can be done in a multilevel regression model.

If your data are reflective of an underlying latent variable, use structural equation modeling or confirmatory factor analysis, and include each variable a manifestation or indicator of your underlying factor.

You bring up the smaller sample size of one of the groups. That imbalance will affect the power of your test (i.e. the ability to detect a difference when it is really there), but shouldn't affect the validity of the tests.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.