Meta-analysis: significant heterogeneity vs. significant between-study variance - Cross Validated most recent 30 from stats.stackexchange.com 2019-08-18T07:10:19Z https://stats.stackexchange.com/feeds/question/242956 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/242956 3 Meta-analysis: significant heterogeneity vs. significant between-study variance Johanna https://stats.stackexchange.com/users/94863 2016-10-28T13:56:25Z 2018-02-01T16:23:32Z <p>I am running a simple random-effects meta-analysis in R using the <em>metafor</em> package, with random intercepts at the study level: </p> <p><code>mod1 &lt;- rma.mv(Hedges_g, cov, random = ~ 1 | study, data = rev)</code></p> <p>This is the model output:</p> <pre><code>Multivariate Meta-Analysis Model (k = 90; method: REML) logLik Deviance AIC BIC AICc -170.3401 340.6802 344.6802 349.6575 344.8197 Variance Components: estim sqrt nlvls fixed factor sigma^2 0.5512 0.7424 24 no study Test for Heterogeneity: Q(df = 89) = 1014.3323, p-val &lt; .0001 Model Results: estimate se zval pval ci.lb ci.ub 0.9749 0.1572 6.2018 &lt;.0001 0.6668 1.2830 </code></pre> <p>As I understand, the observation of significant heterogeneity means that the estimate of g = 0.97 cannot be regarded as an estimate of one <em>true</em> effect. Rather, the studies in this data set seem to be estimating different true effects.</p> <p>Now, I'm comparing my model (mod1) to another model without random intercepts at the study level: <code>mod0 &lt;- rma.mv(Hedges_g, cov, data = rev, method = "ML")</code> (I have set <em>method = "ML"</em> for mod1 too, to enable the comparison). This is the output for <code>anova(mod0, mod1)</code>:</p> <pre><code> df AIC BIC AICc logLik LRT pval QE Full 2 347.2963 352.2960 347.4343 -171.6482 1014.3323 Reduced 1 916.1363 918.6361 916.1818 -457.0682 570.8400 &lt;.0001 1014.3323 </code></pre> <p>Thus, mod1 fits the data significantly better than mod0. This means that the estimated between-study variance of sigma^2 = 0.55, is significant. To me, this would also suggest that the studies are estimating significantly different true effects. </p> <p>My question now is: what is the difference between the test for heterogeneity, and the model comparison? Do they both lead to the exact same conclusion ("There is heterogeneity among the true effects"), or is there more nuance to it?</p> https://stats.stackexchange.com/questions/242956/-/243191#243191 3 Answer by Wolfgang for Meta-analysis: significant heterogeneity vs. significant between-study variance Wolfgang https://stats.stackexchange.com/users/1934 2016-10-30T09:43:56Z 2016-10-30T09:43:56Z <p>Let $\theta_{ij}$ denote the true effect for outcome $j$ in study $i$. The test for heterogeneity given in the output tests the null hypothesis $H_0: \theta_{ij} = \theta$ across all outcomes and studies, that is, whether the true effects are all equal to some common true effect $\theta$.</p> <p>The model you are using (<code>mod1</code>) only allows for heterogeneity in the true effects between studies, not within. Or in other words, it assumes that the true effects within studies are homogeneous. So, let $\theta_{i\bullet}$ denote the true effect for study $i$ that is assumed to be the same for all $j$ outcomes within the study. Then the test you carried out is a test of $H_0 = \theta_{1\bullet} = \ldots = \theta_{k\bullet}$, where $k$ is the number of studies.</p> <p>Assuming homogeneous true effects within studies is a pretty strong assumption that I would not make a priori. Instead, I would suggest to use a three-level model that allows for heterogeneity between and within studies. You can fit this with:</p> <pre><code>rev\$id &lt;- 1:nrow(rev) mod2 &lt;- rma.mv(Hedges_g, cov, random = ~ 1 | study/id, data = rev) </code></pre> <p>See also: <a href="http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011" rel="nofollow">http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011</a></p>