In meta-analysis Q-statistic, is there a better way to compute the degrees of freedom?

Question

I am working with a meta-analysis, and attempting to quantify heterogeneity across several data sources. I am using the following formula:

$Q = \sum_i(w_i \cdot (y_i - \mu_F)^2)$

where $y_i$ is the statistic for that data source, $w_i$ is the weight (which is the reciprocal of the variance of the test statistic), and $\mu_F$ is the weighted mean of the estimates.

According to Higgins 2002, $Q$ should be assumed to be chi-squared distributed with $k-1$ degrees of freedom, where $k$ is the number of data sources I'm using.

However, my data sources neatly divide into approximately twenty whose variances are less than one, and about ten whose variances are greater than ten. This means that some weights are so low (relatively) that they contribute essentially nothing to the weighted mean. Despite this, they add degrees of freedom, which makes the data appear less heterogenous than it would appear if these (very small) sources were omitted.

Is this "good"? i.e. is this a real estimate, that since their confidence intervals are so wide as to include essentially anything plausible, they are "homogenous" with everything?

Or is this "bad", and we are artificially decreasing the heterogeneity measurement? If it is "bad," is there an alternate way to handle this, which is more adaptive to widely disagreeing weights?

Answer 1

If you are interested in quantifying heterogeneity then $Q^2$ is not the best way to do it as it depends on the number of studies which you have. In a paper entitled "Undue reliance on $I^2$ in assessing heterogeneity may mislead" available here Rücker and colleagues outline why $Q^2$ and the commonly used $I^2$ are misleading and suggest using $\tau^2$ instead.