1
$\begingroup$

For a meta-analysis, means and s.d. for the experimental and control groups were required to calculate the mean difference (MD or yi) and variance (vi) of each study with the escalc function of metafor.

In some studies, s.d. needed to be back-calculated from reported p-value and n in experimental (ne) and control (nc) groups (no dispersion measures available).

A problem arise when MD tends toward zero or is null (p-value is always not significant). In such cases, back-calculated s.d. also tends toward zero because MD is the numerator in the formula (see http://handbook.cochrane.org/chapter_7/7_7_3_3_obtaining_standard_deviations_from_standard_errors.htm)

Indeed, very small or null s.d. are not valid when p-value is not significant. Moreover, inclusion of a study with null s.d. in a dataset leads to a warning message in rma.mv: there are outcomes with non-positive sampling variances; and the model doesn't solve.

My question: one approach would be to exclude all studies where back-calculated s.d. is very small or null due to small or null MD. However, these not significant studies contribute to the overall knowledge and should be included somehow in the meta-analysis.

Or should a s.d. be imputed to these studies; e.g., the mean s.d. of the other not significant studies where s.d. could be computed from a dispersion measure (e.g., sem or sed).

Or something else?

$\endgroup$
1
  • $\begingroup$ If the difference is not significant I would expect the standard error (and hence standard deviation) to be large, not small. Can you show us a concrete example of what is going on? $\endgroup$
    – mdewey
    Oct 19, 2016 at 17:41

1 Answer 1

2
$\begingroup$

Without an example of how you are doing it, it's hard to say. I coded up a toy example for myself in R to see what was happening. I could find no values of the mean difference or sample size that would reproduce your problem. Here is some code that you can play around with to confirm that you are doing your calculations correctly.

#back calculating a standard deviation
set.seed(1423)
#data
N = 20 # sample size of each group
truemeandiff = 0.01
# two samples
x1 = rnorm(N, .01, 1)
x2 = rnorm(N, .01-truemeandiff, 1)
meandiff = mean(x1-x2)
(se <- sqrt(var(x1)/N + var(x2)/N))

> 0.3222331

#getting two tailed p-value
z = meandiff/se
(pval.z <- 2*(min(1-pnorm(z), pnorm(z)))) # z-test p-value

> 0.8910154

(pval.t <- t.test(x1, x2, var.equal=TRUE)$p.value) # t-test p-value

> 0.8917388

#back calculating standard deviation from z-test(two-tailed p-value)
abs(meandiff/qnorm(pval.z/2))/(sqrt(1/N + 1/N))

> 1.01899

#back calculating standard deviation from t-test (two-tailed p-value)
abs(meandiff/qt(pval.t/2, N+N-2))/(sqrt(1/N + 1/N))

> 1.01899

# actual standard deviation
se/(sqrt(1/N + 1/N)) 

> 1.01899

Edit: based on the discussion to this answer, you are putting in 0 values for SD, which is not a valid approach to this. A better approach is to back calculate with a substitution of a small non-zero value for the mean difference, as here:

#true mean difference is zero
x1 = rnorm(N, .01, 1)
x2 = x1 
(meandiff <- mean(x1-x2))
#meandiff=0.0001
(se <- sqrt(var(x1)/N + var(x2)/N))

#getting two tailed p-value (works as long as the p-value is not rounded to 1)
fudge = 0.0000001 # making the mean difference small
meandiff = meandiff + fudge
z = meandiff/se
(pval.z <- 2*(min(1-pnorm(z), pnorm(z)))) # z-test p-value
(pval.t <- t.test(x1, x2-fudge, var.equal=TRUE)$p.value) # t-test  p-value

#back calculating standard deviation from t-test (two-tailed p-value)
abs(meandiff/qt(pval.t/2, N+N-2))/(sqrt(1/N + 1/N))

1.19729

# the true sd
se/(sqrt(1/N + 1/N))

1.19729

$\endgroup$
9
  • $\begingroup$ Calculations were correct. However, I am doing a meta-analysis on a dataset of 81 studies. In 12 studies, SD was back-calculated from p-value. Mean difference = 0 and sd = 0 if means of experimental and control groups are the same because meandiff is the numerator in abs(meandiff/qt(pval.t/2, N+N-2))/(sqrt(1/N + 1/N)). Indeed, sd should be large for not significant MD (or MD=0). Inclusion of a single study with sd=0 doesn't solve with rma.mv of library(metafor). $\endgroup$ Oct 20, 2016 at 14:30
  • $\begingroup$ Here is an example: $\endgroup$ Oct 20, 2016 at 15:22
  • $\begingroup$ Example1: study m1 n1 sd1 m2 n2 sd2 1 10 5 2.5 15 5 2.5 2 11 10 3 16 10 3 3 12 8 2.2 13 8 2.2 4 13 7 4 15 7 4 5 14 6 3 14 6 3 example1.md <- escalc(m1i=m1, sd1i=sd1, n1i=n1, m2i=m2, sd2i=sd2,n2i=n2, measure="MD", data=example1, append=TRUE, digits=4) m1 <- rma(yi, vi, method="ML", data=example1.md) summary(m1, digits=3) # results are okay m2 <- rma.mv(yi, vi, method="ML", random = ~ 1|study, data=example1.md) summary(m2, digits=3) # results are okay $\endgroup$ Oct 20, 2016 at 15:29
  • $\begingroup$ Sorry, I tried to include a table of studies but it did not work. There was 5 studies and id of study (1 to 5), means, sd, and n for group 1 and 2 were given in sequence. In any study, if you replace the reported sd1 and sd2 by 0, rma function will not produce I^2 and H^2; and rma.mv will not solve. $\endgroup$ Oct 20, 2016 at 15:36
  • $\begingroup$ Look up dump files for how to give data for working examples with R data. $\endgroup$
    – alex keil
    Oct 20, 2016 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.