2
$\begingroup$

I have been working with the attached data. Using the (wonderful) metafor package I fit a random-effects model, which generates the following outcomes:

rma(yi=yi_diff,vi=vi_diff,data=tempdat)


Random-Effects Model (k = 6; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.3168 (SE = 0.2768)
tau (square root of estimated tau^2 value):      0.5628
I^2 (total heterogeneity / total variability):   79.90%
H^2 (total variability / sampling variability):  4.97

Test for Heterogeneity:
Q(df = 5) = 29.6865, p-val < .0001

Model Results:

estimate      se     zval    pval    ci.lb   ci.ub    
 -0.4189  0.2718  -1.5415  0.1232  -0.9516  0.1137    

---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

As you can see, the p value for the estimated summary effect is non-significant (0.1232).

Leaving aside for a moment the issue of how useful this is, when I run a fail-safe N analysis on this data, I find 2 things.

fsn(yi=yi_diff,vi=vi_diff,data=tempdat)

Fail-safe N Calculation Using the Rosenthal Approach

Observed Significance Level: 0.0002
Target Significance Level:   0.05

Fail-safe N: 22
  1. The p value is different from that returned by the model object.
  2. The fail-safe N is above zero.

This is puzzling to me, because I had understood that the failsafe-N returns the number of studies averaging null results to bring the meta-analytic summary effect to non-significance. Since it is already at a non-significant level, I had expected this value should be zero. Indeed, in the docs for the fsn function, it is written:

If the combined/observed significance level is above the specified alpha level (for type = "Rosenthal" or type = "Rosenberg") or if the observed average outcome is below the target average outcome (for type = "Orwin"), then the fail-safe N value will be 0.

At any rate, it would seem confusing and unhelpful to report that 22 studies are required to reduce an effect that is already non-significant to non-significance!

It appears then, that in addition to all the other known limitations of the fsn, it can also throw up unintuitive results like this. My instinct is that this may have something to do with the different method by which the summary p value is calculated in the fsn analysis as compared to the random effects model. However, surprisingly I am unable to find a clear explanation of conflicts like this. Is anyone able to shed some light on this?

R and metafor version here

$\endgroup$
4
  • $\begingroup$ The fail-safe N has been calculated assuming a fixed-effect model, whereas the the first meta-analysis is based on a random-effects model with the REML estimator of the between-study variance. $\endgroup$
    – Jacob
    Jul 12, 2023 at 13:13
  • 1
    $\begingroup$ @Jacob that looks like an answer to me so perhaps you should post it as such. $\endgroup$
    – mdewey
    Jul 12, 2023 at 14:29
  • $\begingroup$ R is open-source so you can find out what it is doing by looking at the code. Ttyping fsn at the prompt confirms @Jacob comment. The models are different. $\endgroup$
    – mdewey
    Jul 12, 2023 at 14:31
  • $\begingroup$ Thank you @mdewey. I added it as an answer. $\endgroup$
    – Jacob
    Jul 12, 2023 at 19:48

1 Answer 1

1
$\begingroup$

The fail-safe N has been calculated assuming a fixed-effect model, whereas the first meta-analysis is based on a random-effects model with the REML estimator of the between-study variance

$\endgroup$
2
  • $\begingroup$ If this were true then it would seem that the expected behavior should be that a fixed effects meta-analysis should generate the same observed p value as returned by the fsn. It does not. $\endgroup$ Jul 13, 2023 at 12:00
  • $\begingroup$ Check the methods you have used. They are different. The fixed-effect method used in -rma- in the inverse variance method. Rosenthal's approach uses a different fixed-effect method based on the sum of Z scores (not effect sizes). Results should be similar, but not identical, especially if the P-values are <0.01. $\endgroup$
    – Jacob
    Jul 13, 2023 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.