I'm in the process of performing an Structural Equation Modelling (SEM) meta-analysis on some psychological data. Ultimately I want to examine a mediational model based on a set of correlation matrices. I've spent a little time reviewing the literature regarding what to do when you have heterogenous correlation matrices. I've written up a brief review of what I've found on Cognitive Sciences SE.
To summarise what I've found:
- Most researchers use a two-step approach. Step 1 involves extracting estimates of true correlations between a set of variables using a variety of meta-analytic techniques. Step 2 involves inputting the correlation into your SEM software and in general doing analyses as per normal non-meta-analytic SEM.
- Despite an awareness of the problems, many researchers still use the two-step approach even though most datasets have variation in true correlations that can not merely be attributable to random sampling (i.e., a random effects model is generally more applicable).
- Some researchers mention the possibility of clustering the dataset or performing SEM Meta-analysis on values of moderators in the hope that the true score variability will be removed once you divide the analysis up into groups. However, in most applications such moderators are insufficient to explain all the true score variation.
So instead of adopting a two-step approach, I thought about maybe doing the following, at least for cases where you are able to compute a full correlation matrix for each study:
"perform SEM on each sample and treat the parameters and fit statistics as values that vary between samples. You could then summarise the distribution (e.g.,mean and SD) of these SEM parameters and fit statistics. This would be similar to how correlations and other effect sizes are typically modelled as random effects. So, for example you could look at the variation in the indirect effect across samples. The challenging bit would be to tease out what is true score variation and what is due to random sampling."
Excluding the issue of teasing out true score from random sampling, it would be straight forward to do such an analysis in R with any of the SEM packages.
Questions
So, I was wondering:
- Has this approach of fitting SEM to each study separately ever been applied?
- What are the pros and cons of the approach?
- How might you tease out what variation in SEM parameters is due to random sampling and what is due to true score variance?
- Or is there some better approach to doing SEM meta-analysis on heterogenous correlation matrices?
