We used the Q test to determine heterogeneity with p-values <0.10. A highly recognised statistician commented that the Q test has low power and that it should not be used to determine presence of heterogeneity.

My questions:

  • Low power is a known issue. Are there clear alternatives? Can the exact version help in scenarios with low sample sizes?

  • I cannot easily find alternatives and Cochran's Q is standard in the field (his wording was different and quite strong though). Should the Q test not be used to determine presence of heterogeneity? This seems to contradict many other sources. I have seen the test statistic H^2 but it is derived from Cochran’s Q and it is not commonly used (from what I can see).

I have searched across SO sites and elsewhere (Cochrane's Handbook, Meta-Analysis with R from Guido Schwarzer) but can't find clear answers. Related questions are:

Appropriate homogeneity test for meta-analysis

Subgroups Cochran's Q test

Test for homogeneity in meta-analysis with a large number of studies


Two Study Meta-Analysis: Fixed- vs. Random-effects and heterogeneity

Heterogeneity test in meta-analysis

  • $\begingroup$ How many studies/articles you have for your meta-analysis? $\endgroup$ – user158565 Jul 22 '19 at 17:48
  • 1
    $\begingroup$ Hello, we are actually doing a meta-analysis of meta-analyses, so it varies. In some cases two, in others more than ten. $\endgroup$ – antonio Jul 22 '19 at 18:02
  • $\begingroup$ Why do you want to test for heterogeneity? What is the scientific question behind your desire? $\endgroup$ – mdewey Jul 23 '19 at 13:13
  • 1
    $\begingroup$ It's a meta-analysis in epidemiology. We wish to understand how much results vary from study to study. Exploring and testing heterogeneity is part of the standard methods, and usually done with Q tests as part of the statistical exploration (as opposed to eg clinical). $\endgroup$ – antonio Jul 23 '19 at 13:31
  • $\begingroup$ The Q test does not assess heterogeneity. it makes an assessment of homogeneity of effect-sizes. $\endgroup$ – Subhash C. Davar Aug 1 '20 at 12:25

I would just use a likelihood ratio test between your model with and without random effects (on the papers or specifications) where you retain in both models the usual fixed effects for study type etc.

Some cursory discussion is in the article Interpretation of random effects meta-analyses.

Here is step by step instruction:

  1. Estimate a maximum likelihood model, with whatever fixed effects you want to include on study characteristics — usually you will have things like study type (i.e panel vs x section, random trial vs natural experiment), statistical technique (OLS vs logit etc.), sample characteristics (time, sex, age, race, income etc.) and perhaps publication details (time of publication, top-tier journal or not etc.)

  2. estimate as per 1., but add random effects on the papers — you are now estimating a mixed model.

  3. Apply Wilks' D test to test for significance of the random effects.

  4. If (as is almost certain) you find the random effects are significant, you can either use the random effects model, or if you like, use likelihood weighted model averaging over the fixed effects and mixed models.

  • 1
    $\begingroup$ Thanks, not sure I'm following though. Do you mean a ratio between the goodness of fit of e.g. model 1 (random effects) and model 2 (fixed effects)? I think this would give the overall variability but includes sampling, not just differences in treatment effects. Is this right? Would you know of references were this is used? $\endgroup$ – antonio Jul 23 '19 at 13:45
  • $\begingroup$ I will edit to make it clearer. $\endgroup$ – Frustrated Jul 24 '19 at 7:35

The Q-statistic reflects homogeneity statistic and is equivalent to Chi-square statistic. Regarding your question - You can use the Q- statistic to infer presence or absence of homogeneity of effect-sizes. This test checks for heterogeneity indirectly. Note: Box-Pierce test The Box-Pierce Q-statistics are given by:

BP(k)=n∑Kk=1ρ2a,k, [1] where:

ρ2a,k is the autocorrelation coefficient at lag k of the residuals aˆt.

n is the number of terms in differenced series;

K is the maximum lag being considered, set in JDemetra+ to 24 (monthly series) or 8 (quarterly series).

If the residuals are random (which is the case for residuals from a well specified model), they will be distributed as χ2(K−m) degrees of freedom, where m is the number of parameters in the model which has been fitted to the data.

Alternatively, you can execute chi-square test to find out presence of homogeneity in datasets.

  • $\begingroup$ Although Cochran's Q test uses a chi-squared distribution to compute a p-value, that does not make it a "chi-square test." $\endgroup$ – whuber Jul 29 '20 at 14:03
  • 1
    $\begingroup$ Thanks. I had a reading of book - Statistical models for meta analysis by Hedges and Olkin 1985 which points out Q- statistic is equivalent to chi squared statistic. It is my mistake writing as Q test. The second part of answer -the expert statiscian ... is wrong. I shall edit the Answer. I do not know theoretical foundation of Q- statistic. $\endgroup$ – Subhash C. Davar Jul 29 '20 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.