I have a hypothetical research scenario involving multiple comparisons and I want to try and prove my hunch right or wrong! Following a medical intervention, scores for 12 outcome measures are taken after the intervention has ended (T1) and a few months later (T2). Therefore there are 24 comparisons in total, two for each outcome measure compared against the baseline at T1 and again at T2. The researcher then runs 24 t-tests looking for Cohen's d for each one. The results show that three of the comparisons have an effect size of >.8 which was identified beforehand as a reasonable effect size to expect given previous research findings. The researcher takes these three effect sizes as evidence of the intervention's effectiveness and discusses the success accordingly.

I have a suspicion that if the intervention has in fact no true effect on any of the 12 outcome measures, there will be a reasonably high probability of at least three of the 24 comparisons producing effect sizes of >.8 by chance alone. The problem is I can't figure out how to calculate this probability theoretically.

So in general terms that may be useful to others the question is how do I calculate the probability of gaining at least n number of effect sizes greater than or equal to k if I conduct x number of comparisons where there is no true difference between the sample means. Can we calculate this probability apriori if we assume our data will be normally distributed? Many thanks in advance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.