10
$\begingroup$

I am working on a random effects meta-analysis covering a number of studies which do not report standard deviations; all studies do report sample size. I do not believe it is possible to approximate or impute the SD missing data. How should a meta-analysis which uses raw (unstandardized) mean differences as an effect size be weighted when standard deviations are not available for all studies? I can, of course still estimate tau-squared and would like to incorporate that measure of between-study variance in whatever weighting scheme I use to stay within the random-effects framework.

A little more information is included below:

  1. Why raw mean differences might still be useful: The data is reported in a an intrinsically meaningful scale: US dollars per unit. So, a meta-analysis of mean differences would be immediately interpretable.

  2. Why I cannot approximate or impute the SD data: The studies for which standard deviation data is missing do not include enough data to approximate a standard deviation (i.e. median and range are never reported in the literature). Imputing the missing data seems inadvisable as a large portion of the studies are missing the sd, and because the studies differ greatly in terms of geographical region covered and survey protocol.

  3. What is typically done with raw mean differences in meta-analysis: Study weights are based on the standard error of the mean difference (typically computed with sample-size term and the pooled variance). I don't have this. In a random-effects meta-analysis, study weights also include a term for between-study variance. I have this.

Can simple inverse-sample-size weighting be used in this context? How would I incorporate my estimate of tau-squared (or some other measure of between-studies dispersion) into the weighting?

Improve this question
$\endgroup$
2
  • $\begingroup$ If you are prepared to impute your estimate of $\tau^2$ from the studies you have then why are you unwilling to impute values of the standard error? $\endgroup$
    – mdewey
    Commented Feb 23, 2016 at 11:55
  • $\begingroup$ If you do weights based on the sample size, then you assume that the standard deviation of the outcome is exactly the same in all trials. If you think it might vary, it would presumably be better to do something more sophisticated. Also note that US dollars per unit is a problematic scale in that I would expect the variability to be larger for larger mean values. Not sure whether people in your field already have some sensible well-tested way of dealing with that (such as log-transforming or some other sensible approach)? $\endgroup$
    – Björn
    Commented Jun 26, 2017 at 9:50

2 Answers 2

Reset to default
2
$\begingroup$

It would be useful to have more details on your dataset in general, and your meta-analytic estimates in particular. In addition, it would be interesting to know what are the averages and SDs of the complete studies you are including.

Having said that, my pragmatic approach would be, as you hint, to use sample size weighting (why inverse?), but remember this will be at best a hypothesis-generating meta-analysis, whose greatest strength will be pinpointing at the drawbacks of primary studies.

Here are some useful references on the potential use of sample weighting in meta-analysis:

http://faculty.cas.usf.edu/mbrannick/papers/conf/SIOP08Wts.doc

https://www.meta-analysis.com/downloads/Meta%20Analysis%20Fixed%20vs%20Random%20effects.pdf

http://epm.sagepub.com/content/70/1/56.abstract

Improve this answer
$\endgroup$
2
$\begingroup$

If you meta-analyze a mean differences with weights of $n$ instead of by $1/\text{SE}^2$ (inverse variance) - assuming groups of equal size are being compared - this gets you an appropriate average effect estimate under the assumption that variability is the same across studies. I.e. the weights would be proportional to the ones you do would use, if the standard errors were all exactly $2\hat{\sigma}/\sqrt{n}$ for a standard deviation $\sigma$ that is assumed to be identical across trials. You will no longer get a meaningful overall standard error or confidence interval for your overall estimate though, because you are throwing away the information $\hat{\sigma}$ on the sampling variability.

Also note that if groups are not of equal size $n$ is not the correct weight, because the the standard error for the difference of two normal distributions is $\sqrt{\sigma^2_1/n_1 + \sigma^2_2/n_2}$ and this only simplifies to $2\sigma/\sqrt{n}$, if $n_1=n_2=n/2$ (plus $\sigma=\sigma_1=\sigma_2$).

You could of course impute the missing standard errors under the assumption that $\sigma$ is the same across the studies. Then studies without a reported standard error have the same underlying variability as the average of the studies, for which you know it and that's easy to do.

Another thought is that using untransformed US dollars or US dollars per unit might or might not be problematic. Sometimes it can be desirable to use e.g. a log-transformation to meta-analyze and then to back-transform afterwards.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.