2
$\begingroup$

I am working with a meta-analysis, and attempting to quantify heterogeneity across several data sources. I am using the following formula:

$Q = \sum_i(w_i \cdot (y_i - \mu_F)^2)$

where $y_i$ is the statistic for that data source, $w_i$ is the weight (which is the reciprocal of the variance of the test statistic), and $\mu_F$ is the weighted mean of the estimates.

According to Higgins 2002, $Q$ should be assumed to be chi-squared distributed with $k-1$ degrees of freedom, where $k$ is the number of data sources I'm using.

However, my data sources neatly divide into approximately twenty whose variances are less than one, and about ten whose variances are greater than ten. This means that some weights are so low (relatively) that they contribute essentially nothing to the weighted mean. Despite this, they add degrees of freedom, which makes the data appear less heterogenous than it would appear if these (very small) sources were omitted.

Is this "good"? i.e. is this a real estimate, that since their confidence intervals are so wide as to include essentially anything plausible, they are "homogenous" with everything?

Or is this "bad", and we are artificially decreasing the heterogeneity measurement? If it is "bad," is there an alternate way to handle this, which is more adaptive to widely disagreeing weights?

$\endgroup$
2
  • $\begingroup$ $(y_i-\mu_F)^2$? Otherwise Q = 0? $\endgroup$
    – user158565
    Dec 17, 2018 at 23:13
  • $\begingroup$ @user158565 yes, right. I'll edit. $\endgroup$ Dec 18, 2018 at 13:56

1 Answer 1

2
$\begingroup$

If you are interested in quantifying heterogeneity then $Q^2$ is not the best way to do it as it depends on the number of studies which you have. In a paper entitled "Undue reliance on $I^2$ in assessing heterogeneity may mislead" available here Rücker and colleagues outline why $Q^2$ and the commonly used $I^2$ are misleading and suggest using $\tau^2$ instead.

$\endgroup$
2
  • $\begingroup$ That paper suggests using $\tau^2$ as it doesn't depend on the number of studies (good) but it also doesn't depend on the precision of those studies (bad). I'm trying to estimate the heterogeneity between studies, which is not attributable to random noise. $\endgroup$ Dec 20, 2018 at 16:02
  • $\begingroup$ The paper suggests this as a good thing, but since it's intended to be a counter to established industry best practice (using $Q$ and $I^2$ to estimate heterogeneity) I feel it's reasonable to be more critical $\endgroup$ Dec 20, 2018 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.