3
$\begingroup$

I want to run a meta-analysis over several studies reporting one-sample data (e.g. comparing participants' scores against a baseline score of zero). I calculated Cohen's d by dividing the difference of the sample mean and the baseline score by the sample standard deviation (as reported in the first answer here). How do I get the sampling variance of this effect size? (needed in the meta-analysis for calculating inverse variance weights)

$\endgroup$
  • $\begingroup$ I'm using the R package 'metafor' $\endgroup$ – Johanna Aug 2 '16 at 9:57
3
$\begingroup$

This is an interesting question because (so far as I know) there is no widely used formula for computing the variance in this situation. Some time ago, I did some simulations to examine the performance of different formulas to estimate the sampling variance of Cohen's d in case of a one-sample t-test.

I was aware of three different formulas:

The formula used in the Comprehensive Meta-analysis Software:

(1/sqrt(ni))*sqrt(1+di^2/2)^2,

with ni being the sample size per study and di the observed Cohen's d.

Other people use the standard formula for the dependent samples t-test (e.g., Borenstein, 2009) with correlation between pre- and posttest (r) equal to 0.5:

(1/ni)+di^2/(2*ni)

Another formula I have seen is one that was used in a paper by Koenig et al. (2011). This formula is obtained by personal communication with B. Becker.

(1/ni)+di^2/(2*ni*(ni-1)) 

I did a very small simulation study to examine the performance of these three formulas with sample sizes ranging from 10 to 500 and effect sizes in the population ranging from 0 to 0.8. The differences between the formulas were most observable for a population effect size of 0.8.

enter image description here

Using the formula of the dependent samples t-test with r=0.5 yielded the least biased estimates. However, there may be other formulas with better properties. I am curious what other people think about this.

Code:

rm(list = ls()) # Clean workspace

k <- 10000 # Number of studies

thetais <- c(0, 0.2, 0.5, 0.8) # Effect in population 

nis <- c(10,15,20,30,50,75,100,250,500) # Sample size in primary study
sigma <- 1 # Standard deviation in population

### Empty objects for storing results
vi.ac <- vi.beck <- vi.comp <- vi.dep <- matrix(NA, nrow = length(nis), 
                                                ncol = length(thetais), 
                                                dimnames = list(nis, thetais))

############################################
for(thetai in thetais) {
  for(ni in nis) { 

    ### Actual variance Cohen's d
    sdi <- sqrt(sigma/(ni-1) * rchisq(k, df = ni-1))
    mi <- rnorm(k, mean = thetai, sd = sigma/sqrt(ni))
    di <- mi/sdi

    vi.ac[as.character(ni),as.character(thetai)] <- var(di)

    ############################################

    ### Suggestion by Becker in Koenig et al.
    vi <- (1/ni)+di^2/(2*ni*(ni-1))
    vi.beck[as.character(ni),as.character(thetai)] <- mean(vi)

    ############################################

    ### Comprehensive meta-analysis software
    vi <- (1/sqrt(ni))*sqrt(1+di^2/2)^2
    vi.comp[as.character(ni),as.character(thetai)] <- mean(vi)

    ############################################

    ### Dependent sample t-test with r=0.5
    vi <- (1/ni)+di^2/(2*ni)
    vi.dep[as.character(ni),as.character(thetai)] <- mean(vi)

  }
}

plot(x = nis, y = vi.ac[ ,1], type = "l", main = "theta = 0", ylab = "Variance")
lines(x = nis, y = vi.beck[ ,1], type = "l", col = "red")
lines(x = nis, y = vi.comp[ ,1], type = "l", col = "blue")
lines(x = nis, y = vi.dep[ ,1], type = "l", col = "green")
legend("topright", legend = c("Actual variance", "Becker", "CMA", "Dep. samples"), 
       col = c("black", "red", "blue", "green"), lty = c(1,1,1,1))

plot(x = nis, y = vi.ac[ ,2], type = "l", main = "theta = 0.2")
lines(x = nis, y = vi.beck[ ,2], type = "l", col = "red")
lines(x = nis, y = vi.comp[ ,2], type = "l", col = "blue")
lines(x = nis, y = vi.dep[ ,2], type = "l", col = "green")
legend("topright", legend = c("Actual variance", "Becker", "CMA", "Dep. samples"), 
       col = c("black", "red", "blue", "green"), lty = c(1,1,1,1))

plot(x = nis, y = vi.ac[ ,3], type = "l", main = "theta = 0.5")
lines(x = nis, y = vi.beck[ ,3], type = "l", col = "red")
lines(x = nis, y = vi.comp[ ,3], type = "l", col = "blue")
lines(x = nis, y = vi.dep[ ,3], type = "l", col = "green")
legend("topright", legend = c("Actual variance", "Becker", "CMA", "Dep. samples"), 
       col = c("black", "red", "blue", "green"), lty = c(1,1,1,1))

plot(x = nis, y = vi.ac[ ,4], type = "l", main = "theta = 0.8")
lines(x = nis, y = vi.beck[ ,4], type = "l", col = "red")
lines(x = nis, y = vi.comp[ ,4], type = "l", col = "blue")
lines(x = nis, y = vi.dep[ ,4], type = "l", col = "green")
legend("topright", legend = c("Actual variance", "Becker", "CMA", "Dep. samples"), 
       col = c("black", "red", "blue", "green"), lty = c(1,1,1,1))

data.frame(vi.ac[,1], vi.beck[,1], vi.comp[,1], vi.dep[,1])

References:

Borenstein, M. (2009). Effect sizes for continuous data. In H. Cooper, L. V. Hedges & J. C. Valentine (Eds.), The Handbook of Research Synthesis and Meta-Analysis (pp. 221-236). New York: Russell Sage Foundation.

Koenig, A. M., Eagly, A. H., Mitchell, A. A., & Ristikari, T. (2011). Are leader stereotypes masculine? A meta-analysis of three research paradigms. Psychological Bulletin, 137, 4, 616-42.

$\endgroup$
  • $\begingroup$ Thank you, this is extremely helpful! If I want to include your simulation in a paper or presentation, how should I cite you? $\endgroup$ – Johanna Aug 2 '16 at 15:18
  • $\begingroup$ I am happy to help ;-). I think you should just cite this webpage. This simulation is not a an article or book. $\endgroup$ – User33 Aug 4 '16 at 7:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.