# How can one get a non-zero failsafe-N for non-significant meta-analysis?

I have been working with the attached data. Using the (wonderful) metafor package I fit a random-effects model, which generates the following outcomes:

rma(yi=yi_diff,vi=vi_diff,data=tempdat)

Random-Effects Model (k = 6; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.3168 (SE = 0.2768)
tau (square root of estimated tau^2 value):      0.5628
I^2 (total heterogeneity / total variability):   79.90%
H^2 (total variability / sampling variability):  4.97

Test for Heterogeneity:
Q(df = 5) = 29.6865, p-val < .0001

Model Results:

estimate      se     zval    pval    ci.lb   ci.ub
-0.4189  0.2718  -1.5415  0.1232  -0.9516  0.1137

---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


As you can see, the p value for the estimated summary effect is non-significant (0.1232).

Leaving aside for a moment the issue of how useful this is, when I run a fail-safe N analysis on this data, I find 2 things.

fsn(yi=yi_diff,vi=vi_diff,data=tempdat)

Fail-safe N Calculation Using the Rosenthal Approach

Observed Significance Level: 0.0002
Target Significance Level:   0.05

Fail-safe N: 22

1. The p value is different from that returned by the model object.
2. The fail-safe N is above zero.

This is puzzling to me, because I had understood that the failsafe-N returns the number of studies averaging null results to bring the meta-analytic summary effect to non-significance. Since it is already at a non-significant level, I had expected this value should be zero. Indeed, in the docs for the fsn function, it is written:

If the combined/observed significance level is above the specified alpha level (for type = "Rosenthal" or type = "Rosenberg") or if the observed average outcome is below the target average outcome (for type = "Orwin"), then the fail-safe N value will be 0.

At any rate, it would seem confusing and unhelpful to report that 22 studies are required to reduce an effect that is already non-significant to non-significance!

It appears then, that in addition to all the other known limitations of the fsn, it can also throw up unintuitive results like this. My instinct is that this may have something to do with the different method by which the summary p value is calculated in the fsn analysis as compared to the random effects model. However, surprisingly I am unable to find a clear explanation of conflicts like this. Is anyone able to shed some light on this?

R and metafor version here

• The fail-safe N has been calculated assuming a fixed-effect model, whereas the the first meta-analysis is based on a random-effects model with the REML estimator of the between-study variance. Jul 12, 2023 at 13:13
• @Jacob that looks like an answer to me so perhaps you should post it as such. Jul 12, 2023 at 14:29
• R is open-source so you can find out what it is doing by looking at the code. Ttyping fsn at the prompt confirms @Jacob comment. The models are different. Jul 12, 2023 at 14:31
• Thank you @mdewey. I added it as an answer. Jul 12, 2023 at 19:48