Is it worth doing (single-armed) meta analysis if the studies' heterogeneity is very large?

Question

I'm currently trying to perform a single-armed meta analysis of immunotherapy on breast cancer, which I believe this meta analysis will be the first among its intervention and cancer subtype.

The problem is, when I test the heterogeneity using Cochran Q and I-squared statistics, it showed a masive heterogeneity. The I-squared value ranges from 82-90%. Is it worth doing meta analysis with these informations, especially seeing large heterogeneity between studies?

Moreover, when I perform subgroup analysis, 1 parameter showed 0% I-squared value, when the other respectively showed 78.56%, 78.88%, and 90.2%. Is meta analysis still worth doing?

Lastly, if meta analysis is still worth doing: 1. Will the heterogeneity be a problem? 2. Will the review make sense to encourage/discourage the usage of the proposed therapy? 3. Is there any argument(s) that can answer if someone consider the large heterogeneity?

Thank you very much

Answer 1

Single arm studies always involve a comparison to what would have happened without the intervention. If that comparison is done sensibly, then putting the results of these comparisons into a meta analysis may well make sense. Putting some kind of "response" rates from different patient populations (with different expected natural history/placebo response rates) across different follow up periods and possibly further between study differences into a meta analysis and expecting the results to have any meaning that matters to anyone may be illusory. Some things you may be able to (and should) account for, others you will likely be unable to address.

Depending on exactly what was done in each study and what you did with the results would tell you where you are on the continuum from useful to useless.

Answer 2

At the risk of stating the obvious the first thing is to establish what form the heterogeneity takes and whether there is any obvious explanation for it. If the studies are single arm studies they are effectively observational studies. Observational studies are always likely to have more heterogeneous results than trials so readers should not be surprised at the heterogeneity.

One thing worth bearing in mind is the use of $I^2$ here. As Rücker and colleagues have pointed out in an article entitled "Undue reliance on $I^2$ in assessing heterogeneity may mislead" if the individual primary studies have given very precise estimates then high values of $I^2$ are almost inevitable as it compares between study variability to within.

Answer 3

Heterogeneity is NOT a good reason to abandon a meta analysis - rather it is precisely one reason why we would want to do a meta analysis, rather than replication of some extant study with larger sample size.

A meta analysis that finds large heterogeneity will typically give estimates with a far larger SE than the constituent studies, and this finding gives a useful warning to those who would otherwise interpret the small SE in one of the constituent studies as indicating strong support for (or against) some hypothesis.

Now of course if you find a large variance for your random effects, you may want to see if this can be diminished by adding more fixed effects on the study specifications.

If anything, I would find evidence of homogeneity quite troubling - as an indication that the field is immature and only doing one type of study, or more worryingly, trying to hit some 'correct' value it has decided upon theoretically or via deference to past studies. Finding time auto-correlation in the random effects would be a bit of an alarm bell here.