I'm doing an extracurricular programme at my university as part of which we have to write a review article. My supervisor agreed to a meta-analysis. I want to investigate whether the preprocessing of the data (log-transformation of RTs & Outlier trimming/ only outlier trimming/ only log-transformation / raw data) influences the outcome across multiple studies.

For that, I want to analyse all studies on that topic, that my supervisor´s lab ran (ca 10 unpublished + two published papers from other labs) with each of the possible preprocessing pathways. I then planned to conduct a meta-analysis of each pathway separately and compare the mean effect sizes of each preprocessing pathway. How do I account for the fact that each of the 4 meta analyses uses the same studies? Because I think I remember that meta-analysis assumes independence.

Some advice and/or literature would be appreciated. Particularly, if anyone knows a (free) programme that allows me to correct for dependency, that´d be great. And how do I best compare the mean effect sizes of the different meta-analyses? I found a post on here suggesting simply subtracting ES 1 from ES 2. Or should I run a meta-analysis again?

I'm willing to use any free non-programming based program, i.e, preferably not R.

  • $\begingroup$ It is not clear why you want to do meta-analysis if your goal is to compare the effect of the transformations as it seems indirect and throwing away much information. $\endgroup$ – mdewey Mar 12 '19 at 16:21
  • $\begingroup$ A previous paper showed an effect of pre-processing on significance in a sample study. And I thought effect sizes contain more information than p-values. Do you happen to know a more direct approach to synthesize the influence of the pre-processing pathways across studies? $\endgroup$ – Max Primbs Mar 13 '19 at 3:24
  • $\begingroup$ Would it not be more direct to compare the four methods per each study on some metric and then look at them? $\endgroup$ – mdewey Mar 13 '19 at 17:27

Calculating effect sizes does not require running a meta-analysis- meta-analysis is just a way to aggregate effect sizes. If you did need to run a meta-analysis for some reason, though, you would probably need to use something like metafor or metasem in R to account for nonindependence. You would test all effects in a single model and then compare using pathway as a moderator variable and dataset as a grouping variable. Comprehensive Meta Analysis (which is non-programming-based) doesn't handle nested effect sizes, to my knowledge.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.