7
$\begingroup$

My friend and I are trying to understand why a weighted lm() v. a fixed-effect rma model from the metafor package are producing identical meta-analytic point estimates, but different standard errors (and therefore p values and 95% CIs) for those estimates.

For example:

#install and call metafor package
install.packages("metafor")
library(metafor)

#read in example data for standardized mean differences and standard errors
d<-c(0.38, 0.36, -0.35, 1.55, 0.26, 1.2, 0.38, 0.46, 0.27, 0.24, -0.07, -0.26, -0.31, 1.15, 0.23, 0.29, 0.38, 0.19, 0.4, 0.15, 0.2, 0.25, 0.34)
d.se<-c(1.8, 3.49, 1.53, 4.96, 2.08, 3.48, 0.07, 0.07, 0.09, 0.09, 0.01, 0.09, 6.64, 5.08, 7.44, 0.16, 0.18, 2.05, 0.17, 0.16, 0.17, 0.22, 0.09)
d.v<-d.se*d.se

#run fixed-effects intercept-only models with lm() and metafor
lm.intercept<-lm(d ~ 1, weights=I(1/d.v))
summary(lm.intercept)

metafor.intercept<-rma(yi=d, vi=d.v, method="FE")
summary(metafor.intercept)

Both approaches yield the appropriate estimate of -0.0361, but the lm() approach yields a standard error of 0.02585 (and therefore the estimate is not significant at the p >.05 level), whereas the metafor approach yields a standard error of 0.0095 (and therefore the estimate is significant at the p < .05 level). The same discrepancy also occurs for moderators that you add to the model (e.g., d~d.se).

I am somewhat confident that the lm() approach is mistakenly estimating the standard error somehow (I've worked through the calculations for this example by hand: google doc spreadsheet here), but my friend and I would like to better understand why/how this is occurring.

Does anyone have any idea of what's causing the discrepancy in standard errors between the two models?

$\endgroup$
  • 1
    $\begingroup$ The PDF manual for Stata's vwls command contains the answer to your question: stata.com/manuals13/rvwls.pdf See Section "Remarks and examples". $\endgroup$ – boscovich Mar 25 '15 at 20:57
  • $\begingroup$ Thanks @boscovich! Having a friend run model in Stata now to confirm. $\endgroup$ – jsakaluk Mar 25 '15 at 21:13
7
$\begingroup$

A while ago, I wrote up an extensive comparison between the rma() function from the metafor package and the lm() and lme() functions (the latter from the nlme package) for fitting fixed- and random/mixed-effects models. You can find this on the metafor package website:

http://www.metafor-project.org/doku.php/tips:rma_vs_lm_and_lme

To briefly summarize: When you use the lm() and lme() functions with weights, then this fits models that assume that the weights (i.e., sampling variances) are known only up to a proportionality constant -- which is in fact the error variance that is estimated. Those are not standard meta-analytic models as commonly described in the literature.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.