2
$\begingroup$

I have conducted meta-analyses for a Cochrane systematic review. I was comfortable pre-specifying and justifying whether a fixed effect or random effects approach will be used in the MA. Having completed the meta-analyses and used random effects models in some instances my co-author is requesting that I include results from fixed effect models alongside the random effects results.

Given that I pre-stated the analysis approach (although this is an update so there is no registered protocol per-se), then reporting both RE and FE results seems to undermine the purpose of pre-stating the analysis approach.

Should I simply include the FE results, and be clear that they come under sensitivity analysis or stick to my guns and only report what I said would be done? My co-author's view is that FE is more reliable when there is no heterogeneity or there is suspected publication bias.

$\endgroup$
1
  • 2
    $\begingroup$ In my opinion, there is nothing wrong with including the results of FE as sensitivity analysis. It gives indeed more information about the meta-analysis at hand. I would not say that FE is more reliable when there is no heterogeneity in true effect size. The RE model actually reduces to the FE model if there is no heterogeneity. FE assigns less weight to smaller studies if there is heterogeneity, so comparing the results of FE and RE can indeed be insightful if publication bias is present. $\endgroup$
    – User33
    Mar 29 '17 at 19:06
4
$\begingroup$

The debate over FE vs. RE often reflects around the lack of understanding of the concepts behind the models. To make matters worse, the Cochrane handbook only briefly touches on this (9.4.2 Principles of meta-analysis) and provides some guidance on whether or not the posthoc decision to use the FE or the RE was the correct one (10.4.4.1 Comparing fixed and random-effects estimates). From the Handbook: "The combination of intervention effect estimates across studies may optionally incorporate an assumption that the studies are not all estimating the same intervention effect, but estimate intervention effects that follow a distribution across studies. This is the basis of a random-effects meta-analysis (see Section 9.5.4). Alternatively, if it is assumed that each study is estimating exactly the same quantity a fixed-effect meta-analysis is performed."

In only the rarest types of meta-analyses (e.g. same trial repeated multiple times with the same exact protocol) could we assume that the studies are all estimating the same intervention effect, with between-study differences in results attributed to random error. On the other hand, we typically combine studies that use similar populations, similar interventions, similar comparisons, similar doses, etc. assuming that the studies have a reasonable amount of clinical homogeneity allowing their data to be pooled together.

Therefore, IMHO, the decision to use the FE or RE model should be performed at the protocol stage since the actual results do not play a factor in the model choice. Having said that, as noted in section 10.4.4.1 of the Handbook, this choice may need to be reversed after seeing the results of the meta-analysis; if there is suspicion of small-study effect skewing the pooled results.

So, in closing, unless there is justification for the use of FE model, the RE model should be the default choice when conducting a meta-analysis from different study results. Of course, that doesn't mean you can't test test the robustness of the analyses in a sensitivity analysis (FE model). Actually a lot of Cochrane groups request this.

$\endgroup$
1
  • $\begingroup$ This to me is the key point "...the decision to use the FE or RE model should be performed at the protocol stage", but with the pragmatic inclusion of sensitivity analysis involving FE as appropriate. I think this trumps the Cochrane Handbook's recommendation to revise the analysis approach if small-study effects are suspected. $\endgroup$
    – ChrisP
    Mar 30 '17 at 13:52
3
$\begingroup$

To answer this question, one must first consider when the meta analytic estimates from a RE model differ from a FE model. If there is zero between-study heterogeneity, but one uses the RE model, the estimate and interval will be equivalent to a FE model. This is because the FE is assuming 0 heterogeneity. Now, when the estimates differ, this is because there is some degree of variability in the individual effects.

With this in mind, you can ALWAYS assume a RE model, and if there happens to be zero heterogeneity, then your estimates will accurately reflect that fact. The opposite is not true, however, as fixing tau to zero with a FE model "forces" upon your data what is likely an untenable assumption.

With this in mind, if your estimates differ by some amount, this is because there IS between-study variability.

In regards to another answer posted here, I do not see a FE model as a sensitivity analysis. If anything, you will be more likely to reach significance (narrower interval due to possibly underestimating tau). The RE could be considered a sensitivity analysis, but should really just be the default for the above stated reasons.

$\endgroup$
2
  • $\begingroup$ I'm not sure I agree that RE can always be assumed. If FE models are indeed less affected by the biases of small studies, then in this instance it's conceivable that, while both RE and FE are inaccurate, FE may be less inaccurate (although CIs will probably be too narrow). I imagine there is a simulation study which addresses this, but I haven't been able to find one. $\endgroup$
    – ChrisP
    Mar 30 '17 at 13:43
  • $\begingroup$ I'd be interested to see a paper on this topic investigating your notion. It would not be too difficult to simulate yourself. Simply generate some biased effects selected for significance, then answer your question comparing the performance of RE or FE models over the long run. Also, one could take the position they are will to sacrifice some accuracy of the point estimate to ensure they don't erroneously reject the null. I'd be more worried about the ratio of estimate to standard error. Furthermore, if the interval is to narrower, this can increase type one error rate. $\endgroup$
    – D_Williams
    Mar 30 '17 at 15:13
1
$\begingroup$

The distinction here has been ably set out by @abousetta but it is perhaps wrth stressing that neither approach can be regarded as completely satisfactory. The RE estimate is very affected by the smaller studies which are often considered less reliable whereas the FE estimate of the confidence interval is too narrow. The proble has been addressed by Henmi and Copas in an article here entitled "Confidence intervals for random effects meta--analysis and robustness to publication bias". They use the central estimate from the FE model but with a confidence interval which accounts for $\tau^2>0$. The method is available in the metafor package in R. Disclaimer 1: I corrected a few typos in the code and made it available for the package author. Disclaimer 2: I am not claiming that this method is going to solve all the problems of meta-analysis but it is an intriguing option.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.