4
$\begingroup$

In meta-analysis, I know that an important assumption is independence. Clearly, if we want to estimate a pooled effect size, its best if each effect comes from an independent sample of observers. Otherwise, the summary effect will be overly influenced by samples that contribute more than one effect size to the model.

However, as an experimental psychologist, many of the comparisons I am interested in differ at the within subjects level (for instance, the same sample of subjects may have been tested on two sets of stimulus types).

My question, therefore, is whether moderator-analyses can be conducted where k represents conditions, rather than samples. In other words: If I had two effects for every sample, one for stimulus 1 and one for stimulus 2 -could I perform a moderator analysis on stimulus type, given that this varies at the within subjects level?

Note the following things- 1) I wouldn't use this "between condition" model to estimate the "overall" pooled effect size because of the issues I mention in the first paragraph. 2) I have seen published examples of such a "between condition" analysis, but the consequences/ limitations/issues involved are rarely discussed.

Additionally, If anyone has any references/ examples as to why this is ok/ not ok I would be very grateful.

I appreciate your time.

$\endgroup$
2
$\begingroup$

This is possible, though k would not represent your number of conditions; it would still represent the number of studies.

The best way I am aware of testing this kind of moderated hypothesis when you have dependent effect sizes is with Cheung's (2014) three-level multilevel SEM approach. It seems like a more computationally intensive approach--not something I would want to work through by hand--but Cheung's able to explain it conceptually in a pretty straightforward manner, and his metaSEM package for R makes it easy to use in practice.

The syntax looks very similar to other meta-analysis R packages (e.g., metafor); you'll need to specify vectors for your effect sizes (e.g., "effect.size"), variances (e.g., "variance") or standard errors. Then, in the cluster option, you specify the vector that would contain, in your case, article ID numbers. Multiple rows with the same ID would indicate that those effect sizes are nested within the same sample.

For example, to execute a basic random-effects intercept-only model:

library(metaSEM)
intercept<-meta3(y=effect.size, v=variance, cluster=ID, data=dat, model.name="intercept-only model")
summary(intercept) 

Then, for your mixed-effects moderation model, you would specify the vectors for your moderators (e.g., "stimulus") in x, like so:

stimulus.mod<-meta3(y=effect.size, v=variance, cluster=ID, x=cbind(stimulus), data=dat, model.name="moderation by stimulus type")
summary(stimulus.mod)

You could then test whether your moderator model explains a significant amount of variance in effect sizes, above and beyond the intercept only model:

anova(stimulus.mod, intercept)

There are several nice benefits to this approach:

  1. It allows you to accurately estimate a summary effect size while simultaneously and adequately modeling the dependency of your effect sizes (though this estimate might not be all that meaningful if you're expecting different stimulus sets to have polar opposite effects).
  2. Effect size variance is partitioned into occurring at level 2 (e.g., within samples) or level 3 (e.g., between samples). Thus, you can get I-squared estimates that indicate where moderators would be most meaningful. In other words, if most of the variability of your effect sizes is occurring within-samples, between-sample moderators (e.g., year of publication, sample age, etc.,) might not be as important as within-sample moderators.
  3. Given 2., metaSEM will automatically detect whether your specified moderators vary at level 2, level 3 or both. As your "stimulus" moderator likely varies at both levels (e.g., there are some papers that have effect sizes involving one stimulus set or another, whereas other papers might use multiple stimulus sets), metaSEM will therefore estimate how much of the level 2 and level 3 variance in effect sizes is explained by your moderators (i.e., R-squared values for both level 2 and level 3).
  4. Because metaSEM uses an SEM approach to estimating these models, you can specify a number of interesting constraints that otherwise might be very difficult to test. For example, you can test whether the 3-level approach is even necessary (i.e., is there a significant amount of effect size dependency); you can test whether level 2 and level 3 variances are equal; or you could constrain moderator effects to equivalency.

If you're interested in using this approach (I clearly think it's really cool and worthwhile), I highly recommend you check out Cheung's (2014) paper, the website for the metaSEM package, and the help section for metaSEM on the OpenMx website (metaSEM uses the OpenMx package). Cheung also has a book coming out soon in which he details this approach more thoroughly.

References

Cheung, M. W. L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19, 211-229.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.