Meta-analysis and homogeneity -- what did these guys do?

Question

I appreciate any insight into this meta-analysis.

This is a meta-analysis on alogliptin efficacy and safety. In the 2nd paragraph of the discussion that make this statement:

Although heterogeneity testing showed a statistically significant dissimilarity in the results of the included studies, sensitivity analysis has shown the stability of the overall odds ratios with the withdrawal of any of the study from the analysis without a significant improvement of the heterogeneity. Thus, the credibility of the results of this meta-analysis did not seem compromised. This is because; when the number of included studies is small and heterogeneity is large, the robustness of the results is best assessed with a sensitivity analysis

What in the world do they mean by "sensitivity analysis has shown the stability of the overall odds ratios with the withdrawal of any of the study from the analysis without a significant improvement of the heterogeneity"

Do they mean when one outlier study is removed the ORs don't change? What are they trying to communicate? Please help!

Citation: Efficacy of alogliptin in type 2 diabetes treatment: a meta-analysis of randomized double-blind controlled studies (PDF) - BMC Endocrine Disorders 2013, 13:9

Answer 1

One of the meta-analytic techniques for sensitivity analyses is known as "one study removed" and it simply means that. What effect does each single included study have on the overall effect estimate. I haven't had a chance to look at the paper but can tell you from the description is that the authors don't fully understand the issue of statistical heterogeneity or how to deal with it. You can't just that all my studies are across the board therefore everything is fine and let's pool. You need to be methodical throughout the whole process and checking the effect of each study on the overall heterogeneity is just one step. First they need to make their data is correctly extracted and inputted. #1 cause of heterogeneity is wrong data (e.g. extracting SE instead of SD). If the data is valid, then they need to check differences at each step in the PICOTSS (especially for clinical heterogeneity). If none there, then comes the statistical sensitivity analyses (e.g. removing one study at a time, unclear/ high risk of bias trial vs. low risk of bias trials, funding sources, etc.). In the end, you may still not find a single source of heterogeneity. In this case, you have to make a judgment call on whether or not to present pooled results or just go with a descriptive analysis (most investigators like to pool).

Hope this helps.

Ahmed Abou-Setta, MD, PhD

Answer 2

The more sophisticated underlying problem is this - are the apparent study level or specification level random effects approximately normal.

Now consider the following Hypotheses:

(1) There are no paper/specification level random effects - all of the variance in the estimates across studies is a result of within study errors or fixed effects on study characteristics.

(2) There are paper/specification level random effects, and they are well represented by a single normal distribution

(3) There are paper/specification level random effects, and they are not well represented by a single normal distribution.

Now if (3) is the case, one particular problem will be if there is large excess kutosis. In this case, random effects in the extreme tails will occur with higher frequency than under normally distributed random effects.

Now the kludge way to do this is to simply remove the 'outliers' and see if the results change dramatically.

The better way to do this is to explicitly model non-normal random effects. There are a few promising ways to do this:

(a) use some single non-normal distribution

(b) Use multiple distributions, with random assortment

(c) use multiple distributions, with assortment via some identifiable characteristics