I am working with a meta-analysis, and attempting to quantify heterogeneity across several data sources. I am using the following formula:

$Q = \sum_i(w_i \cdot (y_i - \mu_F)^2)$

where $y_i$ is the statistic for that data source, $w_i$ is the weight (which is the reciprocal of the variance of the test statistic), and $\mu_F$ is the weighted mean of the estimates.

According to Higgins 2002, $Q$ should be assumed to be chi-squared distributed with $k-1$ degrees of freedom, where $k$ is the number of data sources I'm using.

However, my data sources neatly divide into approximately twenty whose variances are less than one, and about ten whose variances are greater than ten. This means that some weights are so low (relatively) that they contribute essentially nothing to the weighted mean. Despite this, they add degrees of freedom, which makes the data appear less heterogenous than it would appear if these (very small) sources were omitted.

Is this "good"? i.e. is this a real estimate, that since their confidence intervals are so wide as to include essentially anything plausible, they are "homogenous" with everything?

Or is this "bad", and we are artificially decreasing the heterogeneity measurement? If it is "bad," is there an alternate way to handle this, which is more adaptive to widely disagreeing weights?

  • $\begingroup$ $(y_i-\mu_F)^2$? Otherwise Q = 0? $\endgroup$ – user158565 Dec 17 '18 at 23:13
  • $\begingroup$ @user158565 yes, right. I'll edit. $\endgroup$ – Richard Rast Dec 18 '18 at 13:56

If you are interested in quantifying heterogeneity then $Q^2$ is not the best way to do it as it depends on the number of studies which you have. In a paper entitled "Undue reliance on $I^2$ in assessing heterogeneity may mislead" available here Rücker and colleagues outline why $Q^2$ and the commonly used $I^2$ are misleading and suggest using $\tau^2$ instead.

  • $\begingroup$ That paper suggests using $\tau^2$ as it doesn't depend on the number of studies (good) but it also doesn't depend on the precision of those studies (bad). I'm trying to estimate the heterogeneity between studies, which is not attributable to random noise. $\endgroup$ – Richard Rast Dec 20 '18 at 16:02
  • $\begingroup$ The paper suggests this as a good thing, but since it's intended to be a counter to established industry best practice (using $Q$ and $I^2$ to estimate heterogeneity) I feel it's reasonable to be more critical $\endgroup$ – Richard Rast Dec 20 '18 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.