I am doing a meta-analysis and frequently need to calculate the effect sizes from the data since the effect size I am looking for is frequently not reported directly.

I am looking at a paper where groups of people made an evaluation before and after an intervention. (Multiple subjects in a working together in a trial; multiple such trials with the intervention are held with different people each time). There is a "correct" evaluation and we can calculate the proportion of people in each trial who got the correct evaluation both before and after the intervention.

For e.g. there are N (say 20, but its not always the same) people in a trial. They first individually evaluate a stimulus. Some proportion get it right and some proportion get it wrong (let's call this proportion who got it right Prop1). Then all the people in the trial undergo the intervention and we ask people to re-evaluate the stimulus. This time Prop2 % of people get it right. Therefore, the claim is that the intervention changed the proportion from Prop1 to Prop2. We assume that this causal link is not in question for the purposes of this question. (I am looking to get an effect size for this intervention.)

I have calculated using Cohen's d, and D-rm (as mentioned here https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00863/full), but the effect size is too big compared to everything else I have and this leads me to believe that the error is in the way I'm calculating it. The mean proportion of people who tend to evaluate correctly jumps from 55% to 62% so my hunch is the effect size cant be that big.(I could be way off on this front)

Is the paired sample test the best way to do this? Are there other methods / stats I need to be looking at? I have looked at Effect size and bootstrapping in paired t-test post previously but it doesn't seem to match what I want either.

Would be greatly thankful for any help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.