Hoping I can find some help.

I'm writing the methods section to a systematic review and I want to mention my approach to comparing Treatment Outcomes and Clinical Efficacy.

The title of the systematic review is: A systematic review into the efficacy of interventions utilising semantics in the treatment of bilingual Aphasia.

So basically I'm reporting on 12 selected research studies and the efficacy of a specific intervention type as mentioned in these studies.

4/12 articles report effect sizes which I can compare and interpret from established criteria but the remaining 8 either report McNemar X2/report ANOVA/T-test results/simply report pre/post intervention % score differences.

My thought at this stage is to try and use the data available in the non-effect-size reporting articles/convert other measures so I have 12 effect sizes to compare...problem is, I have no idea how to do this and don't know if I have sufficient data available to me to even do this as the research papers when not reporting effect sizes don't give me mean data in two groups and don't provide me with standard deviation scores. So I don't have data sets to work with.

What should I do?

  • $\begingroup$ It is not that simple. Need to review 12 studies carefully. At first, to check what response variables they used. Secondly, how they expressed the efficacy; it can be difference between treatment, ratio of trt A vs trt B, or difference of difference. Then come back to effect size. $\endgroup$
    – user158565
    Commented Jul 27, 2019 at 18:28

2 Answers 2


Sorry I didn't have the characters to post my original response. Sorry it got fragmented above.

Essentially I've been following the structure layed out by two other systematic reviews. They obviously faced the same challenges and dealt with it in different ways. If you read these my thinking will hopefully be clearer.

1 - Each study was examined for the question(s) which it addressed and relevant pre- and post-therapy data were extracted.We computed statistical significance for the pre- and post- treatment scores using the McNemar’s change test (p < 0.05, Seigel & Castellan,1988).3 There were two primary reasons for performing the statistical computations. Some studies failed to report any statistical measure (e.g., Faroqi & Chengappa, 1996; Gil & Goral, 2004; Khamis, Venkert-Olenik, & Gil, 1996). A few other studies reported parametric statistical tests whose assumptions of normality and independence were not met by the study design and data. McNemar’s change test is a non-parametric test for paired nominal measures such as accuracy data, and has been used by several aphasiologists to compute statistical significance of treatment-induced changes in behavioral scores (Faroqi-Shah, 2008; Rochon, Laird, Bose, & Scofield, 2005). The use of a consistent statistical measure makes comparisons of statistical significance across studies more valid.

2 - As well as describing the treatment outcomes of included studies, the clinical efficacy of SFA was determined by calculating effect sizes. Effect sizes could be calculated only in those studies that reported sufficient data. To calculate, it was necessary to determine the individual values for the pretreatment and posttreatment phases for each set of trained items. Cohen’s d statistic was used to calculate effect size as described by Busk and Serlin (1992). The magnitude of change in performance was determined according to the benchmarks for lexical retrieval studies described by Beeson and Robey (2006). The benchmarks were 4.0, 7.0, and 10.1 for small, medium, and large effect sizes, respectively. Where Cohen’s d could not be calculated, the percent of nonoverlapping data (PND) was calculated. PND is the most widely used method of calculating effect size in single case experimental designs (Gast, 2010; Schlosser, Lee, & Wendt, 2008). PND is the percentage of Phase B data points (the treatment phase) that do not overlap with Phase A data points (baseline or no treatment). To determine the magnitude of effect, benchmarks put forth by Scruggs, Mastropieri, and Casto (1987) were used. PND scores higher than 90% were considered to demonstrate a highly effective treatment, PND of 70%–90% were interpreted as a moderate treatment outcome, and PND scores of 50%–70% were considered a questionable effect. PND scores less than 50% were interpreted as an ineffective intervention because performance during intervention had not affected behavior beyond baseline performance.

So as you can see, it makes sense from these that I need to create similar measures to compare efficacy between studies.


Are you interested in giving an overview of the results in these 12 papers? In that case it would be as simple as just reporting how many of the 12 'found' a hypothesized effect and how much didn't, I guess?

If you want to actually combine the 12 results into a single result, I think you will be very hard pressed if results were reported in different ways and without the underlying data..

  • $\begingroup$ Thanks for reply and your time. Well I was following the structure layed out in two other systematic reviews around a similar topic (not the exact same focus). $\endgroup$ Commented Jul 27, 2019 at 18:33
  • $\begingroup$ One explains there approach in this way: ''We computed statistical significance for the pre- and post-treatment scores using the McNemar’s change test (p < $\endgroup$ Commented Jul 27, 2019 at 18:34
  • $\begingroup$ I'm not very well versed with these types of studies, but it seems as if these other systematic reviews have obtained the actual lab data from these studies? Or otherwise, I presume they have made some effort to retrospectively simulate the data based on the results in the papers? I think without it would be very difficult to make the type of claims mentioned in the two papers.. $\endgroup$ Commented Jul 27, 2019 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.