Meta-analysis with zero values for mean difference and s.d

Question

For a meta-analysis, means and s.d. for the experimental and control groups were required to calculate the mean difference (MD or yi) and variance (vi) of each study with the escalc function of metafor.

In some studies, s.d. needed to be back-calculated from reported p-value and n in experimental (ne) and control (nc) groups (no dispersion measures available).

A problem arise when MD tends toward zero or is null (p-value is always not significant). In such cases, back-calculated s.d. also tends toward zero because MD is the numerator in the formula (see http://handbook.cochrane.org/chapter_7/7_7_3_3_obtaining_standard_deviations_from_standard_errors.htm)

Indeed, very small or null s.d. are not valid when p-value is not significant. Moreover, inclusion of a study with null s.d. in a dataset leads to a warning message in rma.mv: there are outcomes with non-positive sampling variances; and the model doesn't solve.

My question: one approach would be to exclude all studies where back-calculated s.d. is very small or null due to small or null MD. However, these not significant studies contribute to the overall knowledge and should be included somehow in the meta-analysis.

Or should a s.d. be imputed to these studies; e.g., the mean s.d. of the other not significant studies where s.d. could be computed from a dispersion measure (e.g., sem or sed).

Or something else?

Answer 1

Without an example of how you are doing it, it's hard to say. I coded up a toy example for myself in R to see what was happening. I could find no values of the mean difference or sample size that would reproduce your problem. Here is some code that you can play around with to confirm that you are doing your calculations correctly.

#back calculating a standard deviation
set.seed(1423)
#data
N = 20 # sample size of each group
truemeandiff = 0.01
# two samples
x1 = rnorm(N, .01, 1)
x2 = rnorm(N, .01-truemeandiff, 1)
meandiff = mean(x1-x2)
(se <- sqrt(var(x1)/N + var(x2)/N))

> 0.3222331

#getting two tailed p-value
z = meandiff/se
(pval.z <- 2*(min(1-pnorm(z), pnorm(z)))) # z-test p-value

> 0.8910154

(pval.t <- t.test(x1, x2, var.equal=TRUE)$p.value) # t-test p-value

> 0.8917388

#back calculating standard deviation from z-test(two-tailed p-value)
abs(meandiff/qnorm(pval.z/2))/(sqrt(1/N + 1/N))

> 1.01899

#back calculating standard deviation from t-test (two-tailed p-value)
abs(meandiff/qt(pval.t/2, N+N-2))/(sqrt(1/N + 1/N))

> 1.01899

# actual standard deviation
se/(sqrt(1/N + 1/N)) 

> 1.01899

Edit: based on the discussion to this answer, you are putting in 0 values for SD, which is not a valid approach to this. A better approach is to back calculate with a substitution of a small non-zero value for the mean difference, as here:

#true mean difference is zero
x1 = rnorm(N, .01, 1)
x2 = x1 
(meandiff <- mean(x1-x2))
#meandiff=0.0001
(se <- sqrt(var(x1)/N + var(x2)/N))

#getting two tailed p-value (works as long as the p-value is not rounded to 1)
fudge = 0.0000001 # making the mean difference small
meandiff = meandiff + fudge
z = meandiff/se
(pval.z <- 2*(min(1-pnorm(z), pnorm(z)))) # z-test p-value
(pval.t <- t.test(x1, x2-fudge, var.equal=TRUE)$p.value) # t-test  p-value

#back calculating standard deviation from t-test (two-tailed p-value)
abs(meandiff/qt(pval.t/2, N+N-2))/(sqrt(1/N + 1/N))

1.19729

# the true sd
se/(sqrt(1/N + 1/N))

1.19729