1
$\begingroup$

I have got a question regarding the comparison of the effect size comparing a new (digital) intervention with previous data using a traditional intervention. In summary, we've designed a new (digital - smartphone based) intervention and a suited (digital) placebo - intervention. We conducted an online study comparing the pre and post scores of randomly assigned participants (103 participants completed intervention, 180 completed placebo). We used a mixed-model ANOVA with the factors TIME (pre, post) and the factor GROUP (treatment, placebo) and could show a significant interaction effect.

So far, so good. Now we want to set our results in context to classical experiments previously done in our lab. We've got similar data for the classical treatment vs. its placebo-procedure of 4 separate studies (20 vs. 20, 26 vs. 26, 20 vs. 20, 50 vs. 50 participants).

How can I compare these previous data with our new data? We want to show that our new digital intervention works comparably good as the old one.

I thought, I could do similar ANOVAs for the 4 older studies separately and compare the effect sizes (partial eta squared). But I could also only compare the intervention groups, doing paired t-tests for each study's intervention group and compare the effect sizes (cohen's d). I was also thinking of pooling all participants together and put them into one model. What would be the best (most correct) way to answer my question?

Thanks in advance, I appreciate it.

$\endgroup$
1
  • $\begingroup$ @mdewey's answers is perfect... just an additional hint, simply add study id as additional random effect to your mixed anova model (or a similar mixed effect linear model) and you are done... $\endgroup$ Commented Jun 14, 2021 at 7:52

1 Answer 1

2
$\begingroup$

If you have the individual data then you can fit a single model to the whole dataset. This is usually called individual participant data meta-analysis if you do it with many different data sources. Sometimes also called one-step IPD to distinguish it from the other option you mention, to analyse each set separately then meta-analyse them, known as two-step meta-analysis.

If you have individual-level covariates then one-step is clearly better otherwise there should be little practical difference.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.