I am conducting multilevel meta-analyses in R using the metafor package. I have 177 treatment vs. control comparisons from the data from 30 studies (multiple rows per study due to multi-year studies). Thus I am using random effects model rma.mv by keeping study id (reference or citation name) and year (the year in which observation was taken) as random effects (random=~1|id/year). I also want to see fixed effects of two moderators (crop and soil). Moderator “crop” has 7 levels and moderator “soil” has 3 levels. I ran first model (Model1) using rma.mv function, to see the overall effect of treatment. Then I ran three other models to test the effect of two moderators (please see syntax). The QM test is significant only for Model2 and Model4. Then I compared all four models using AIC criteria, that showed Model1 (and Model2?) is the best among all, however it does not include any moderator. AIC for Model1, 2, 3 and 4 are -77.07, -75.82, -56.95 and -58.28, respectively.

My questions are:

  1. If I choose Model1, according to AIC, then how could I test and discuss the effects of crop and soil types on response variable?
  2. When I use rma function, the results differ compared to that when I use rma.mv, in terms of significance. Which function should I choose?

Thank you in advance.

Model1<-rma.mv(yi, vi, random=~1|id/year, data=b)

Model2<-rma.mv(yi, vi, mods = ~ factor(soil)- 1, random=~1|id/year, data = a)
Model3<-rma.mv(yi, vi, mods = ~ factor(crop) - 1, random=~1|id/year, data = a)

Model4<-rma.mv(yi, vi, mods = ~ factor(soil)+factor(crop) - 1, random=~1|id/year, data = a)

1 Answer 1


Comparisons of information criteria only make sense under maximum likelihood estimation. [1] So if you are going to compare AIC values, fit the models with method="ML".

Also, you remove the intercept term in models 2, 3, and 4. This does not tell you if the factor is statistically significant (see: https://www.metafor-project.org/doku.php/tips:models_with_or_without_intercept). If you want to test if there are differences between the levels of a factor, leave the intercept in the model.

As for your second question: rma.mv() since your data has a multilevel structure, which rma() would not account for.

[1] There is actually some support for possibly comparing information criteria under REML estimation (https://doi.org/10.1198/000313006X90396). However, this is a contentious issue and to what extent the results from Gurka (2006) transfer to other types of models remains to be examined.

  • $\begingroup$ Thanks Dr. Wolfgang. Now I have modified my model as : $\endgroup$
    – grad011
    Jun 9, 2021 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.