13
$\begingroup$

Description of the study:

I have observed a common error among meta-analyses, with regard to handling of within-study replication. It is not clear to me if the error invalidates the studies when assumptions are stated. However, as I understand it, these assumptions violate a basic premise of statistics.

As an example, a study tests the effects of chemical $X$ on response $Y$.

The analysis is performed on the log response ratio: the ratio of treatment $Y_{+X}$ (in the presence of $X$) to control $Y_{0}$ (no $X$):

$$R = \ln(\frac{Y_{+X}}{Y_{0}})$$

Some of the studies included in the meta-analysis contain multiple treatments, for example different levels or chemical forms of $X$. For each treatment, there is a different value of $R$, although $R$ always uses the same value of $Y_0$.

The methods state:

responses to different treatments (levels and forms of $X$) within a single study were considered independent observations.

Questions:

  • Isn't this pseudoreplication?
  • Is it inappropriate even if the violation of independence is stated in the methods?
  • What would be an easy way (e.g. within the ability of a simple meta-analysis software package) to handle within study replication?

Initial thoughts:

  • Summarize results of each study, e.g. by taking the average response
  • Select only one treatment from each study based on a-priori criteria (e.g. highest dose, first measurement)?

Are there any other solutions?

Improve this question
$\endgroup$
4
  • $\begingroup$ This is just a quick guess but you might want to check Kim/Becker 2010: The Degree of Dependence between Multiple-Treatment Effect Sizes; I haven't read the article but it might be related to your question. $\endgroup$
    – Bernd Weiss
    Commented Oct 5, 2011 at 20:31
  • $\begingroup$ Is the meta-analysis really just averaging all these difference values of R? That seems rather strange, compared to e.g. attempting a meta-regression - in which case the differences between R at different levels of X might be what you're interested in combining across studies. $\endgroup$
    – guest
    Commented Jan 24, 2012 at 6:50
  • $\begingroup$ @guest yes, they really are; it would be of interest how different levels of X affect R, but the question is posed simply "is there an effect of X"? There may be limited power to test the effect of X on R, in this context (ecosystem response to nutrient addition), because of the variety of methods and study conditions. $\endgroup$
    – David LeBauer
    Commented Jan 24, 2012 at 15:26
  • 1
    $\begingroup$ You're right, it is a problem. Not so much with the point estimates, but the measures of precision (i.e. standard errors) will be too small; it ignores the multiple use of the control group's data. It shouldn't however, be news to anyone in meta-analysis. The Kim/Becker article above is basically a re-statement -with acknowledgement- of Gleser & Olkin (1994). Stochastically dependent effect sizes. In Cooper & Hedges (Eds), The handbook of research synthesis (pp. 339–355). This book is a standard text in the field, I believe now in a second edition. $\endgroup$
    – guest
    Commented Jan 25, 2012 at 5:12

2 Answers 2

Reset to default
3
$\begingroup$

Yes, it is a problem because there is sampling dependence in the responses which needed to be accounted for (although sometimes the effect might be negligible and we do violate assumption all the time when we perform statistical analyses). There are methods to deal with this, one approach is to include the covariances between related experiments (off-diagonal blocks) in the error variance-covariance matrix (see e.g. Hedges et al., 2010). Fortunately with log ratios this is rather easy. You can get approximated covariances between experiments because the variance (var) of log R is (if Yx and Y0 are independent groups): log Yx - log Y0, to follow the notation in the question, Yx referring to the experimental group and Y0 the control group. The covariance (cov) between two values (e.g. treatment 1 och treatment 2) for log R is cov(loge Yx_1 - log Y0, log Yx_2 - log Y0), which equals var(log Y0), and is calculated as the SD_Y0/(n_Y0 * Y0), where SD_Y0 is the standard deviation of Y0, n_Y0 is the sample size in the control treatment, and Y0 is the value in the control treatment. Now we can plug-in the whole variance-covariance matrix into our model instead of only using the variances (ei) which is the classic way to perform a meta-analysis. An example of this can be found in Limpens et al. 2011 using the metahdep package in R (on bioconductor), or Stevens and Taylor 2009 for Hedge´s D.

If you want to keep it very simple, I would be tempted to ignore the problem and try to evaluate the effect of sampling dependence (e.g. how many treatments are there within studies? how do the results change if I only use one treatment? etc) .

Improve this answer
$\endgroup$
2
$\begingroup$

Yes, this is a problem.

Yes, it is inappropriate even though at least it is transparent about what it is doing (it gets points for transparency, but still is not satisfactory).

I doubt there is an "easy way" to fix this. I don't know much about the approaches taken to meta analysis but if there is specific meta-analysis software and research like this is produced using it and gets published, this may well be the common approach. Either of your proposed responses loses some granularity of information from each study (ie the opposite problem of what the publishers have done).

The obvious solution is a mixed-effects (ie multilevel) model with study as a random factor. I would suggest using a specialist statistical package for this if meta-analysis software cannot do it. You could still use the meta-analysis software for data storage and processing, and just export data to R, Stata or SAS for the analysis.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I had been thinking about clinical trials and wondering if it was okay in the situation where a dose-response curve was the outcome, because then it could be comparing curve functions. Is that a possibility? $\endgroup$
    – Michelle
    Commented Jan 29, 2012 at 19:36
  • $\begingroup$ I don't think it makes much difference to the problem that several results from one study will somehow be correlated and hence not "new" information. But meta analysis of curve functions would certainly be possible, so long as you somehow controlled for correlation between the various estimates of those curves. If they're all of the same form and it is just a matter of estimating parameters it should be possible. $\endgroup$
    – Peter Ellis
    Commented Jan 30, 2012 at 23:19
  • $\begingroup$ @Michelle I concur with Peter: if you are summarizing the parameters of the curve, then you get one parameter estimate from each curve, and that should be fine. $\endgroup$
    – Abe
    Commented Feb 13, 2013 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.