2
$\begingroup$

We used the Q test to determine heterogeneity with p-values <0.10. A highly recognised statistician commented that the Q test has low power and that it should not be used to determine presence of heterogeneity.

My questions:

  • Low power is a known issue. Are there clear alternatives? Can the exact version help in scenarios with low sample sizes?

  • I cannot easily find alternatives and Cochran's Q is standard in the field (his wording was different and quite strong though). Should the Q test not be used to determine presence of heterogeneity? This seems to contradict many other sources. I have seen the test statistic H^2 but it is derived from Cochran’s Q and it is not commonly used (from what I can see).

I have searched across SO sites and elsewhere (Cochrane's Handbook, Meta-Analysis with R from Guido Schwarzer) but can't find clear answers. Related questions are:

Appropriate homogeneity test for meta-analysis

Subgroups Cochran's Q test

Test for homogeneity in meta-analysis with a large number of studies

https://stackoverflow.com/questions/56362186/exact-cochrans-q-test-in-r?r=SearchResults

Two Study Meta-Analysis: Fixed- vs. Random-effects and heterogeneity

Heterogeneity test in meta-analysis

$\endgroup$
  • $\begingroup$ How many studies/articles you have for your meta-analysis? $\endgroup$ – user158565 Jul 22 at 17:48
  • 1
    $\begingroup$ Hello, we are actually doing a meta-analysis of meta-analyses, so it varies. In some cases two, in others more than ten. $\endgroup$ – antonio Jul 22 at 18:02
  • $\begingroup$ Why do you want to test for heterogeneity? What is the scientific question behind your desire? $\endgroup$ – mdewey Jul 23 at 13:13
  • 1
    $\begingroup$ It's a meta-analysis in epidemiology. We wish to understand how much results vary from study to study. Exploring and testing heterogeneity is part of the standard methods, and usually done with Q tests as part of the statistical exploration (as opposed to eg clinical). $\endgroup$ – antonio Jul 23 at 13:31
  • $\begingroup$ Hello, we are actually doing a meta-analysis of meta-analyses, so it varies. In some cases two, in others more than ten. what is meta-analysis of meta-analyses? $\endgroup$ – Subhash C. Davar Jul 28 at 12:10
2
$\begingroup$

I would just use a likelihood ratio test between your model with and without random effects (on the papers or specifications) where you retain in both models the usual fixed effects for study type etc.

Some cursory discussion is in the article Interpretation of random effects meta-analyses.

Here is step by step instruction:

  1. Estimate a maximum likelihood model, with whatever fixed effects you want to include on study characteristics — usually you will have things like study type (i.e panel vs x section, random trial vs natural experiment), statistical technique (OLS vs logit etc.), sample characteristics (time, sex, age, race, income etc.) and perhaps publication details (time of publication, top-tier journal or not etc.)

  2. estimate as per 1., but add random effects on the papers — you are now estimating a mixed model.

  3. Apply Wilks' D test to test for significance of the random effects.

  4. If (as is almost certain) you find the random effects are significant, you can either use the random effects model, or if you like, use likelihood weighted model averaging over the fixed effects and mixed models.

$\endgroup$
  • 1
    $\begingroup$ Thanks, not sure I'm following though. Do you mean a ratio between the goodness of fit of e.g. model 1 (random effects) and model 2 (fixed effects)? I think this would give the overall variability but includes sampling, not just differences in treatment effects. Is this right? Would you know of references were this is used? $\endgroup$ – antonio Jul 23 at 13:45
  • $\begingroup$ I will edit to make it clearer. $\endgroup$ – Frustrated Jul 24 at 7:35
0
$\begingroup$

Q-test is another name for chi-square test. It is not a test of heterogeniety.The Q-test or chi-square test geneates homogeneity statistic. The expert statistician has correctly suggested that it should not be used to determine presence (or absence of heterogeneity).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.