Let’s say we have an experiment at one location evaluating the effect of some sort of treatment on a control condition (n=4). We calculate the effect size as (Treated – Control)/Control*100. So for example:
set.seed(123) treated.1 <- c(rnorm(n=4, mean=100, sd=10)) control.1 <- c(rnorm(n=4, mean=50, sd=5)) effect.size.1 <- (treated.1-control.1)/control.1*100 t.test(effect.size.1) # Effect size = 110, 95% C.I = 56 - 166%, df = 3
We then add a location. To continue the example:
treated.2 <- c(rnorm(n=4, mean=80, sd=8)) control.2 <- c(rnorm(n=4, mean=70, sd=7)) effect.size.2 <- (treated.2-control.2)/control.2*100 t.test(effect.size.2) # Effect size = 11, 95% C.I = -17 - 39%, df = 3
What do we do though if we want the average effect size across the two sites? There seems to be at least two ways to approach this. We could simply go on as before and combine the replications without regard to site:
# Approach 1 effect.size.ind <- c(effect.size.1, effect.size.2) t.test(effect.size.ind) # Effect size = 61%, 95% C.I = 11 - 110%, df = 7
Or we could figure the average effect size for each site first:
# Approach 2 effect.size.avgs <- c(mean(effect.size.1), mean(effect.size.2)) t.test(effect.size.avgs) # Effect size = 61%, 95% C.I = -573 - 696%. df = 1
The approach doesn’t affect the estimated effect size (+61%), but it does affect the confidence intervals (+11 to 110% vs. -573 to 696%) through its effect on the degrees of freedom (7 vs. 1).
I’ve seen studies calculating effect size with multiple reps at one site and then meta-analysis studies calculating effect size with average effect sizes from different sites, but I haven’t seen a study looking at effect size with multiple reps at multiple sites. So I'm not sure how this is normally addressed.
So my question is two-fold:
- What are the disadvantages/advantages/interpretations of each of the two approaches above?
- Is there a better or more appropriate way to go about this?