Heterogeneity measures when pooling meta-analyses: what assumptions do we make?

Question

I am looking at the possibilities and limitations of meta-meta-analysis (MMA), put differently, pooling the results of multiple meta-analysis. I realise that it is better to include primary studies, but that is not my point here, I'm trying to find out what happens if you do a MMA.

My current issue is about calculating heterogeneity in the MMA. By doing a few tests, I realised that the I2 of the MMA would be different depending on in what way the studies are combined in the MAs. So, example, if we have two MAs that included A, B and C for MA1, and D, E, and F for MA2, the I2 of the MMA can be (very) different, compared to when MA1 included studies A, C, E, and MA2 included B, D, F. I'm not surprised by this, but it made me wonder how this happens and what influences the result. Is the I2 of the MMA even an interpretable number or doesn't it say anything about the results? I assume that the I2 is worth something if there is no heterogeneity in each of the primary MAs. Correct?

Are there any other assumptions that we (must) make and check when doing a MMA? Does anybody have any recommendations for literature on this topic?

If it is relevant, I'm looking at it from an epidemiological point-of-view.

Answer 1

Here is one way to solve your problem, if the data is good enough to avoid collinearity:

(1) add fixed effects on the inclusion of this or that paper, where the default is the common set of papers (those papers which appear in all MA). The fixed effect estimates will be an indication of the impact that a certain paper has in shifting the MA results. If your data is not good enough to support fixed effects, then consider using random effects. Both of these techniques account for variability in results across MA that result from the inclusion of a different set of papers.

(2) add random effects on the MA specification, if a Wilks' D test suggests you should include them. This is a reasonably good test for heterogeneity. Due to step (1) it a test specifically of heterogeneity that results from different researchers using a different method of aggregating results, not just from using different papers.

(3) attain your central estimate and error bars under the condition of the inclusion of all papers.