5
$\begingroup$

While reading "Conducting Meta-Analyses in R with the metafor Package" (http://www.jstatsoft.org/v36/i03/paper), I am wondering how model fitting works with user defined weights. In chapter 2.4 is written:

Next, $\mu$ or $\beta_0, \dots, \beta_{p'}$ are estimated via weighted least squares with weights equal to $w_i = 1/(v_i + \tau^2)$, where $\tau^2$ denotes the estimate of $\tau^2$.

Holds this for user defined weights? In detail: Is $\tau^2$ then part of the estimation of $\mu$? If not, how to incorporate an artifact cleaning factor like the attenuation factor from Hunter&Schmidt?

$\endgroup$
5
$\begingroup$

If you specify user-defined weights, then those are used for estimating $\mu$ (which is just the weighted average of the observed effects or outcomes using those weights). That is, after all, the entire point of being able to specify user-defined weights. The same idea applies when fitting (mixed-effects) meta-regression models (where we estimate $\beta_0, \ldots, \beta_p$), but I'll keep things simple in my answer by just focusing on the random-effects model here.

While some descriptions of the random/mixed-effects model seem to suggest that inverse-variance weights (i.e., $w_i = 1/(v_i + \tau^2)$) is the only possible option for the weights to use, this is not so. The estimate of $\mu$ is unbiased (assuming that the effect size estimates are unbiased) even if you specify any other arbitrary weights.

Under the model, the inverse-variance weights have the advantage of giving you the uniformly minimum-variance unbiased estimator (UMVUE) of $\mu$ (again, assuming that the effect size estimates are unbiased), that is, among all unbiased estimators, it is the most efficient estimator. However, that is only true if you would actually know the true values of the sampling variances (i.e., the $v_i$ values) and $\tau^2$ (the variance of the true effects, or as typically called in this context, the amount of heterogeneity). In practice, we compute estimates of the $v_i$ values (typically based on large-sample approximations) and we estimate $\tau^2$ (with one of the dozen or so estimators available for this purpose) and then just use those estimates to construct the inverse-variance weights. In that case, we do not really get the UMVUE anymore though.

So, one could easily consider alternative (used-defined) weights that may, in theory, be not quite as efficient, but that still work very well in practice and that may have other advantages. Let me give you an example:

One form of publication bias involves the suppression of non-significant findings from the published literature. Since smaller studies (i.e., studies where the sampling variance is large) with results closer to the null value will tend to be non-significant, their suppression will lead to positive bias in the estimate of $\mu$ (i.e., we will tend to get an overestimate). This problem is exacerbated by using the usual inverse-variance weights, especially when the amount of heterogeneity is large. In that case, the estimate of $\tau^2$ will tend to be large and the weights will tend to be quite similar to using equal/unity weights, so the small studies (where the bias is more severe) get almost as much weight as the large studies (which are less affected by the publication bias). By using, for example, weights equal to $w_i = 1/v_i$, we give more weight to the large studies and thereby diminish the biasing influence of the small studies.

Note that while $\tau^2$ is then not part of the weights, the amount of heterogeneity still influences the precision of the estimate of $\mu$. That is still correctly taken into consideration when the standard error of the estimate of $\mu$ is computed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.