Torn between PET-PEESE and multilevel approaches to meta-analysis: is there a happy medium?

Question

I am currently working on a meta-analysis, for which I need to analyze multiple effect sizes nested within samples. I am partial to Cheung's (2014) three-level meta-analysis approach to meta-analyzing dependent effect sizes, as opposed to some of the other possible strategies (e.g., ignoring dependency, averaging effect sizes within studies, selecting one effect size, or shifting the unit of analysis). Many of my dependent effect sizes are correlations involve fairly distinctive (but topically related) variables, so averaging across them does not make conceptual sense, and even if it did, it would cut my number of total effect sizes to analyze by nearly half.

At the same time, however, I am also interested in using Stanley & Doucouliagos's (2014) method of addressing publication bias in the course of estimating a meta-analytic effect. In a nutshell, one either fits a meta-regression model predicting study effect sizes by their respective variances (the precision effect test, or PET), or their respective standard errors (the precision effect estimate with standard errors, or PEESE). Depending on the significance of the intercept in the PET model, one either uses the intercept from the PET model (if the PET intercept p > .05) or the PEESE model (if the PET intercept p < .05) as the estimated publication-bias-free mean effect size.

My problem, however, stems from this excerpt of Stanley & Doucouliagos (2014):

In our simulations, excess unexplained heterogeneity is always included; thus, by conventional practice, REE [random-effects estimators] should be preferred over FEE [fixed-effects estimators]. However, conventional practice is wrong when there is publication selection. With selection for statistical significance, REE is always more biased than FEE (Table 3). This predictable inferiority is due to the fact that REE is itself a weighted average of the simple mean, which has the largest publication bias, and FEE.

This passage leads me to believe that I should not be using PET-PEESE in random-effects/mixed-effects meta-analytic models, but a multilevel meta-analytic model would seem to require a random-effects estimator.

I am torn as to what to do. I want to be able to model all of my dependent effect sizes, but simultaneously take advantage of this particular method of correcting for publication bias. Is there some way for me to legitimately integrate the 3-level meta-analysis strategy with PET-PEESE?

References

Cheung, M. W. L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19, 211-229.

Stanley, T. D., & Doucouliagos, H. (2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods, 5, 60-78.

Answer 1

I have worked on a meta-analysis following mainly the Cheung approach (but not the 3 levels) and recently came across the PET-PEESE approach for correcting publication bias. I was also intrigued in combinations of the two approaches. So far my experience. I think there are two ways to tackle your problem. A simple one and a more complicated one.

The quote below seems to suggest that random effects exacerbate the publication bias so to me it seems that if you suspect publication bias to be an issue, you cannot simply use a random effects model.

With selection for statistical significance, REE is always more biased than FEE (Table 3). This predictable inferiority is due to the fact that REE is itself a weighted average of the simple mean, which has the largest publication bias, and FEE.

I am assuming that publication bias is a serious concern.

Simple approach: Model the heterogeneity under PET-PEESE

If I understood the questions correctly, I think this approach is the most pragmatic starting point.

The PET-PEESE approach lends itself to extensions to meta-analytic regressions. If the source of heterogeneity stems mainly from the different variables in the effect sizes than you can model the heterogeneity as fixed effects by including indicator variables (1/0) for each variable*. In addition, if you suspect that some variables have better measurement properties or are closer related to your construct of interest, you might want to have a look at the Hunter and Schmidt style of meta-analyis. They propose some corrections for measurement error.

This approach would probably give you an initial idea of the size of the publication bias via the PET and PEESE intercepts and of the heterogeneity based on the variance in the fixed effects.

The more complicated approach: Model heterogeneity and publication bias explicitly

I mean that you explicitly model the occurrence of publication bias according to the Stanley and Doucouliagos paper. You also have to explicitly write out the three levels of Cheung as random effects. In other words, this approach requires you to specify the likelihood yourself and would probably be a methodological contribution in itself.

I think it is possible to specify such a likelihood (with appropriate priors) following a hierarchical Bayes approach in Stan and use the posterior estimates. The manual has a short section on meta-analysis. The users list is also very helpful.

The second approach is probably overkill for what you want at this stage but it would probably be more correct than the first approach. And I would be interested in whether it works.

* If you have a lot of variables (and not a lot of effect sizes) than it might be better to group similar variables into groups (yes, that is a judgement call), and use group indicator variables.