Reporting of Heterogeneity in Metaanalysis

The background is a forest plot of a meta-analysis which is reported to have been calculated with a "random effects model". The pooled effect is reported as a standardized mean difference (SMD).

Results below the forest plot are listed to have:

Heterogeneity: I-squared=94%, tau-squared=0,11, p<0,0002

1. Regarding the reporting of heterogeneity:

• a) What is the practical value of the reported heterogeneity in regard to the interpretation of the results of a meta-analysis? (In my understanding it shows the differences of the included study results in relation to the overall results)
• b) Is there, for example, a certain "threshold percentage" when results should not be considered as reliable or meta-analyis should not be undertaken at all?
2. Regarding the reported significance (p<0,0002)

• a) Does it refer to the heterogeneity results or the pooled effect results of the overall forest plot? (In my understanding it refers only to the heterogeneity results.)
• b) What is the practical value of the reported significance in regard to interpretation? (Does it somehow show the validity or robustness of the heterogeneity results?)

The question is asked from a beginners perspective in statistics.

More background information: the metaanalysis was done with six different studies, also using different outcome measures. No subgroup analysis was possible because of heterogeneity. Meta-regression ist also not advisable when there are less than 10 studies, according to the Cochrane foundation http://handbook.cochrane.org/chapter_9/9_6_4_meta_regression.htm

• Have you tried a subgroup analysis or a meta-regression to explain part of that heterogeneity? – GGA Jan 30 '16 at 14:32
• I did not do the calculations myself. I only received the forest plot. I was informed that a subgroup analysis is not possible because of the hetereogeneity of the studies. – Sebastian Jan 30 '16 at 15:14
• In my (very basic) statistic understanding the low number of included studies means that a meta-regression is not possible. – Sebastian Jan 30 '16 at 15:57
• 1 a ) is not clear ? Shows the differences of .... – Subhash C. Davar Aug 6 '17 at 9:21

Those are all interesting questions. I will label them 1a, 1b, 2a, 2b.

1a the $\tau^2$ shows you how variable the true effect sizes are, ie it is implying that there is indeed a distribution of true effect sizes rather than just a single one. So you are correct if I understand you well.

1b that depends on what you want to do. If you want one single estimate it implies you are in rouble, if your goal is to understand what is going on it has told you something (see 1a).

2a Yes, you are correct

2b if the studies involved are all very precisely estimated (perhaps because they are large) then it is virtually bound to be significant. If you only have a few imprecise studies it is almost sure not to be. So it is not that interesting without further thought and examination of what is going on.

Disclaimer: this is a personal view, some people would not agree with me on 1b.

• Ans. 2a may be substantiated to establish a link between heterogeneity and p statistic. – Subhash C. Davar Aug 3 '17 at 15:17
• Ans 2a and 2b are incorrect. – Subhash C. Davar Dec 12 '17 at 4:34