Heterogeneity test in meta-analysis

Question

In a random effect meta-analysis model with one categorical independent variable $\theta_{ij}=\theta_i+s_j+\epsilon_j$, where $\theta_{ij}$ is the observed effect size in category $i$ in study $j$, $\theta_i$ is the mean effect size in category $i$, $s_j$ is the random study effect, and $\epsilon_j$ is the within study error (or sampling error). We often further assume that $s_i$ follows a normal distribution with mean 0 and variance $\tau^2$. $\epsilon_j$ is also normally distributed and its variance ($\sigma^2$) is calculated from primary literature and is assumed to be known when fitting the model.

In a model like this, meta-analysis software, such as metafor in R, can perform two heterogeneity test. The first is testing if among study heterogeneity is 0, that is, $\tau=0$. This is done by computing a Cochran's $Q=\sum w_i(\theta_{ij}-\hat{\theta_i})^2$. Here, $w_i=1/\sigma^2$ and $\hat{\theta_i}$ is the weighted average ($w_i$ as the weight) of all $\theta_{ij}$ within the $i$th group. Meta-analysis text book often says that Q is asymptopically chi-square distributed under the null hypothesis $\tau=0$.

The second homogeneity test is whether all $\theta_i$ are equal. This is done by computing a $Q=\sum w^*_i(\hat{\theta_i}-\hat{\theta})^2$, where $\hat{\theta_i}$ is the weighted ($w^*_i=1/(\hat{\tau^2}+\sigma^2)$) average of all $\theta_{ij}$ within the $i$he category and $\hat{\theta}$ is the weighted overall mean. Again, Q follows a Chi-square distribution.

The questions I have are

1. In the first test, Why is Q asymptotically chi-square, not exactly Chi-square? We know for a normally distributed random variable, $(n-1)S^2/\sigma^2$ follows chi-square distribution. Q is essentially the same. Is it the weighted average that makes it asymptotically, but not exactly, Chi-square?

2. Is the heterogeneity test for $\tau$ still valid in general for a linear mixed effect model? In a linear mixed model, we usually don't know $\sigma$ and have to estimate it using "ML" or "REML". Does this break the Chi-square distribution of Q? I have never seen the Q statistics for testing random effect in a linear mixed model text book. We know that likelihood ratio or Wald test for random effect tends to be conservative. If this Q statistics works in general, why isn't it adopted as a general test for linear mixed models?

3. For the second heterogeneity test about the categorical predictor, is it just a Wald test? We know that Wald test or likelihood ratio test are anti-conservative in linear mixed model. Many text book recommends conditional F test (with df corrections) or simulation based inference. Is such anti-conservative nature still a concern for meta-analysis model? I don't know whether the fact that $\sigma$ is given to the model as a known quantity alleviate such problem.

I have read a few meta-analysis textbook and search online but cannot find detailed explanations for these technical details. Any insights or references would be very helpful.

Answer 1

I have done more reading on primary literature since posting the questions. I want to post what I have learnt since then and hopefully that might be helpful for someone else. I cannot guarantee that these new findings are absolutely correct. So please feel free to comment and correct any wrong understandings here.

1. If we treat $\sigma$ as known, $Q$ follows an exact Chi-square distribution. The proof is similar to proving $(n-1)S^2/\sigma^2$ follows Chi-square distribution. Using weighted average in the calculation does not break the Chi-square distribution. However, in practice, $\sigma$ is not actually known. It is calculated based on data in primary literature and treated as known. This is what makes the distribution asymptotic. This 1982 paper by Hedges seems to be the first paper that address the distribution property of the Q statistic.

2. It only works in the settings of meta-analysis because $\sigma$ is treated as known. In linear mixed model, $\sigma$ is unknown and has to be estimated. If we use the estimated $\sigma$ in calculating Q, the Q statistics does not have Chi-square distribution any more. In fact, if $\sigma$ is estimated in the absence of any random effect (i.e. in a linear model), Q is actually always n-1.

3. It is indeed a Wald test. And Wald test tends to be anti-conservative in linear mixed model. Because Wald test is based on asymptotic theory, sample size influences its performance. I think this is why the anti-conservative problem is less severe in meta-analysis. In a meta-analysis, the random effect $s_j$ is for each $\theta_{ij}$ and thus we have a large sample size to estimate $\tau$. For example, if we compiled 100 effect size estimates for a meta-analysis without any moderator, we have 99 degrees of freedom to estimate $\tau$. In a normal linear mixed model, we often don't have that many samples to estimate random effect. For example, we may do an experiment in a few randomly chosen fields or conduct an experiment in several randomly chosen schools. This makes the anti-conservative property of the Wald test a more severe concern in general for linear mixed model. However, such anti-conservative problem is recognized in meta-analysis literature as well. Adjustments such as the Knapp-Hartung method has been proposed to at least partially resolve this issue.