2
$\begingroup$

My newbie question is how to select the best model including different types of moderator variables in fixed or mixed models. Here's some example data (from the package help):

data(dat.bcg)

### meta-analysis of the log relative risks using a random-effects model
res0 <- rma(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)

### mixed-effects model with absolute latitude as a moderator
res1 <- rma(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, mods = ~ ablat, data=dat.bcg)

### mixed-effects model with two moderators (absolute latitude and publication year)
res2 <- rma(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg,
           mods = ~ ablat + year, data=dat.bcg)

### mixed-effects model with two moderators (one of which is a factor)
res3 <- rma(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg,
           mods = ~ ablat + factor(alloc), data=dat.bcg)

Is it right to select the model simply using Akaike Information Criterion?

> AIC(res0, res1, res2, res3)
#      df      AIC
# res0  2 28.40474
# res1  3 22.17464
# res2  4 24.21375
# res3  5 24.79656

Finally, selecting model res2 for the lowest AIC value.

$\endgroup$

1 Answer 1

1
$\begingroup$

The answer must depend on what your scientific hypothesis was as the three predictors you mention are of different types. Distance from the equator (ablat) was suggested as a possible influence on the effectiveness of the vaccine so has a real scientific interest. Publication year (year) might suggest that the effectiveness was changing over time or more likely that studies were becoming better controlled. Allocation to treatment type (alloc) is a study quality variable. If you pick the model with the lowest AIC you risk missing important facts about the vaccine.

$\endgroup$
7
  • $\begingroup$ I understand your point, but I only used that example without considering the meaning of the moderators... I wonder which is the best model in statistical terms. $\endgroup$
    – Juanchi
    Commented Sep 5, 2016 at 17:10
  • 2
    $\begingroup$ @Juanchi What exactly do you mean by "best model in statistical terms"? $\endgroup$
    – Wolfgang
    Commented Sep 5, 2016 at 18:37
  • 2
    $\begingroup$ The "best" model should not be confused with the other models not being useful. Moreover, imagine a significant effect of whatever on the odds of developing a disease. However, this models AIC difference between lowest AIC is only 1.95 lower (2 as the general guideline). I would hope that AIC would not exclusively guide your inference in this situation. $\endgroup$
    – D_Williams
    Commented Sep 5, 2016 at 18:53
  • $\begingroup$ @Wolfgang In my real dataset I have 4 potential moderator variables and I don't really know how to select the final model for a best estimation of the effect sizes. $\endgroup$
    – Juanchi
    Commented Sep 5, 2016 at 19:26
  • 1
    $\begingroup$ Have you seen this? metafor-project.org/doku.php/tips:model_selection_with_glmulti Note also the comment by @D_Williams - so you may want to use multimodel inference. $\endgroup$
    – Wolfgang
    Commented Sep 6, 2016 at 5:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.