I need to conduct a meta-analysis for a publication, but this is my first meta-analysis and I still don’t feel confident. I will describe the steps I have followed, and hopefully some of you might find errors on my methods, and suggest alternatives.

For this particular analysis, long-term studies should be more important than short-term because a few experiments showed transient effects, hence short-term studies might fail to capture that the effect is not really significant in the long-term. On my dataset, about half the studies have several non-independent measurements taken at different time-points (i.e. several annual measurements). Other experiments, despite having been carried out for several years, show the data already aggregated, with only one row per study with mean and standard deviation. I considered running a multivariate meta-analysis to solve this issue, but it would unbalance the analysis, giving more importance to the experiments with several rows of data (annual measurements) than to the experiments with aggregated data in only one row (pooled across several years). Am I right? This is an example of the dataset:

  • Study 1, Year 1, Effect Size 1
  • Study 1, Year 2, Effect Size 2
  • Study 1, Year 3, Effect Size 3
  • Study 2, Year 1, Effect Size 4
  • Study 3, Years 1-4, Effect Size 5
  • Study 4, Years 1-3, Effect Size 6

Alternatively, I decided to try and aggregate the data, so that finally there is only one row per study. I followed these steps:

Calculate effect sites for each row, including those studies with several rows (annual data). In this case, I calculated the log response ratio (ROM):

dat <- escalc (measure="ROM”, n1i=elev.rep, n2i=control.rep, m1i=elev.ANPP.mean, m2i=control.ANPP.mean, sd1i=elev.SD, sd2i=control.SD, data=all)

Aggregate studies using the function agg {MAd}. I used the Borenstein et al. 2009 method, and correlation=1:

datAgg <- agg(id = id,es = yi,var = vi, cor =1,method = "BHHR", data = dat)

I have now only one row per study. However, since long-term experiments are more important, I have created user-defined weights that take into account the number of replicates and the number of years of each study:

datAgg$weightsTime <- with(datAgg, ((control.rep * elev.rep)/(control.rep + elev.rep)) + ((nyears^2)/(2*nyears)))

Run the mixed-effects meta-regression with two moderators, using Hedges Estimator (HE) and the Knapp and Hartung approach:

m <- rma.uni(yi, vi, mods= ~ factor(A) * factor(B), method="HE", data=datAgg, weights=weightsTime, knha=TRUE)

Am I doing something wrong? Can this method be improved? So far the results confirm my hypothesis, but of course I might be using a sub-optimal approach. Many thanks


1 Answer 1


I agree that 'multivariate meta-analysis' in this case is hard to justify, particularly if there is a possibility of repeated measures effect (however, in that case, why would it be reported on an annual basis, was there any indication that the samples may have been independent? this would be insightful to explore). In particular, rma.uni has an assumption of 'independent studies'.

Based on your justification, it seems reasonable to use weights for the length of the study (although it would help if you justify why you use those particular weights). Yet, an alternative approach, if your sample size allows, would be to do separate analyses for those two categories (log and short-term) to avoid information loss.

Weights that take into account the number of replicates are open to debate, however, because if a study had sufficient power, it is not entirely defensible to down-weigh it due to lower sample size (the model is not accounting for within-study variability). Also note that, unless you have very long studies (unlikely) or very few reps, your reps penalty is worse/stronger than the length of study penalty, which I think is unfortunate.

The agg part seems well-supported. rma.uni has the same assumptions as GLM, so if satisfied, it should be ok.

  • $\begingroup$ Thanks for your comment. In mi field (climate change ecology) means + SD are usually presented in figures on an annual basis (e.g. plant growth over time due to water treatment) to analyse trends, but I wouldn’t say those measurements are independent. I can justify my weighting function based on other papers (e.g. de Graaff et al. 2006; van Groenigen et al. 2006), but I agree with you that using the inverse of the variance would be better. However, in my dataset only 2 studies out of 90 are assigned more than 50% of the total weight, so w=1/var is not ideal. $\endgroup$
    – fede_luppi
    Oct 15, 2015 at 8:39
  • $\begingroup$ "plant growth over time", ok then, for sure. I did not actually suggest inverse variance; I see that de Graaff et al. 2006 weight is somewhat justified, but if I were reviewing this, I'd still like to see both scenarios, with and without the n reps weighting, for reasons stated above, as well as explicit information on what went into those weights (ranges of sample sizes, ranges of duration). $\endgroup$
    – katya
    Oct 15, 2015 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.