I have performed meta-analysis and included four different tests for publication bias in the analysis - classic fail-safe N, Orwin's fail-safe N, Duval and Tweedie's trim and fill as well as funnel plots. The tests did not seem to indicate publication bias in my material - according to classic fail-safe N 23 papers would be needed to bring the result to p > 0.05, according to Orwin's fail-safe N 8 papers would be needed to reduce the OR into a trivial one, trim and fill test did not show adjustments and funnel plot was symmetrical.

However, one of my reviewers wants me to include cut-offs for publication bias tests in the Methods. I have tried to look for any cut-offs that would be commonly used with these tests but in many cases even the interpretation of results is not very exact. How should I proceed with this problem, what should I answer to the reviewer?



The use of any fail-safe N method is depreciated. Please see Becker (2005) and check out the Conclusions section. Here, she states:

"Given the other approaches that now exist for dealing with publication bias, the fail-safe N should be abandoned in favour of other, more informative analyses [...]".

You also might want to read her statement in Summary of the Examples:

"The incredible range of values shown here reveals one of the greatest weaknesses of the failsafe N computations - it is difficult to interpret the values without a statistical criterion."

The "trim and fill method" is a good starting point. However, you also might want to give Egger's regression method a try which, for instance, is described in Sterne/Egger (2005) (please note that Egger et al. (1997) suggest a weighted least squares regression, while Sterne and Egger (2005: 101) recommend a simple OLS regression). See also this CV question on Egger’s linear regression method intercept in meta analysis.

For a general overview of methods to detect publication bias see Sutton et al. (2000) or these slides by Birgit Schrödle.


Egger, M., G. Davey Smith, M. Schneider, C. Minder (1997), Bias in meta-analysis detected by a simple, graphical test British Medical Journal 315: 629-634.

Sterne, J.A.C., M. Egger (2005), Regression Methods to Detect Publication and Other Bias in Meta-Analysis. Pp. 99-110 in: H.R. Rothstein, A.J. Sutton, M. Borenstein (eds.), Publication Bias in Meta-Analysis. Prevention, Assessment and Adjustments, The Atrium, Southern Gate, Chichester: Wiley.

Sutton, A.J., F. Song, S.M. Gilbody, K.R. Abrams (2000), Modelling publication bias in meta-analysis: a review. Statistical Methods in Medical Research 9: 421-445

  • $\begingroup$ Thank you for the prompt and very useful comment! I will proceed with the Egger's regression method. $\endgroup$
    – Annie
    Feb 8 '12 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.