What kind of cut-offs are used for publication bias tests in meta-analysis?

Question

I have performed meta-analysis and included four different tests for publication bias in the analysis - classic fail-safe N, Orwin's fail-safe N, Duval and Tweedie's trim and fill as well as funnel plots. The tests did not seem to indicate publication bias in my material - according to classic fail-safe N 23 papers would be needed to bring the result to p > 0.05, according to Orwin's fail-safe N 8 papers would be needed to reduce the OR into a trivial one, trim and fill test did not show adjustments and funnel plot was symmetrical.

However, one of my reviewers wants me to include cut-offs for publication bias tests in the Methods. I have tried to look for any cut-offs that would be commonly used with these tests but in many cases even the interpretation of results is not very exact. How should I proceed with this problem, what should I answer to the reviewer?

Thanks!

Answer 1

The use of any fail-safe N method is depreciated. Please see Becker (2005) and check out the Conclusions section. Here, she states:

"Given the other approaches that now exist for dealing with publication bias, the fail-safe N should be abandoned in favour of other, more informative analyses [...]".

You also might want to read her statement in Summary of the Examples:

"The incredible range of values shown here reveals one of the greatest weaknesses of the failsafe N computations - it is difficult to interpret the values without a statistical criterion."

The "trim and fill method" is a good starting point. However, you also might want to give Egger's regression method a try which, for instance, is described in Sterne/Egger (2005) (please note that Egger et al. (1997) suggest a weighted least squares regression, while Sterne and Egger (2005: 101) recommend a simple OLS regression). See also this CV question on Egger’s linear regression method intercept in meta analysis.

For a general overview of methods to detect publication bias see Sutton et al. (2000) or these slides by Birgit Schrödle.

References

Egger, M., G. Davey Smith, M. Schneider, C. Minder (1997), Bias in meta-analysis detected by a simple, graphical test British Medical Journal 315: 629-634.

Sterne, J.A.C., M. Egger (2005), Regression Methods to Detect Publication and Other Bias in Meta-Analysis. Pp. 99-110 in: H.R. Rothstein, A.J. Sutton, M. Borenstein (eds.), Publication Bias in Meta-Analysis. Prevention, Assessment and Adjustments, The Atrium, Southern Gate, Chichester: Wiley.

Sutton, A.J., F. Song, S.M. Gilbody, K.R. Abrams (2000), Modelling publication bias in meta-analysis: a review. Statistical Methods in Medical Research 9: 421-445