I need a little help or reassurance concerning how to explain power "to power", i.e. to decision makers that are not well versed in statistics.
The problem is this: I have done three empirical studies on a treatment and the effect on an outcome. The treatment ranges in value from small to fairly large. There is no natural way to pool the treatments, since they differ in terms of target population, period of study and "internal treatment logic", i.e. what is done to the treated. The target outcome is the same for all studies. You can think of it as giving some classrooms extra math lessons,
Study 1 - 40 more hours
Study 2 - 20 more hours
Study 3 - 10 more hours
All other classrooms - 0 more hours
And then measuring the improvement in a test score for math skills across the whole school.
For all studies, I use a matching approach (coarsened exact matching) on covariates pre-treatment, and then a weighted multiple linear regression (weights are proportions of treated to untreated in matching strata). I find effect sizes of between 5 and 20 per cent compared with an untreated population (pupils in all other classrooms). (
Study 1 is sufficiently powered - large number of treated (large class, no "new" pupils), good matching - and those give me type-I significant estimates of about 15 per cent. This could be expected and 15 percent is a reasonable result, since this is the "large" treatment.
The two other studies are more problematic. Study 2 has few treated (but a reasonable matching and number of controls), but I estimate an effect size of almost 20 per cent, not statistically significant (p=0.14)
Study 3 has an effect size of around 8 per cent, not significant (p=0.39) and a large variance in the outcome
Both studies 2 and 3 are underpowered, using an guesstimated effect size of maybe 10 percent for study 2 and maybe 5 percent for study 3. at $\alpha=0.9$ and $\beta=0.7$ (don't ask for a rationale for these numbers, they just seem reasonable... )
Now I need to summarize this over "increased math hours program".
I would like to say is that study 1 is "generalizable", i.e. I can use study to guide expectation and expect the outcome to be positive. I cannot say that for study 2 and 3 - in those cases the results are valid for the pupils studied. I feel like an undergraduate statistics student again, but is this an accurate description of the inference?
- I expect a positive effect on math score for all untreated classrooms with 40 extra math hours
- I would not have a reasonable guess as to the effect of giving less than 40 hours (20 or 10 hours). A bigger study with more pupils is needed to capture smaller expected effect sizes.
- All programs improve average math scores for the pupils in the study
I need some reassurance, because some stakeholders (teachers?) interpret insignificant as "no effect whatsoever" and I am beginning to believe them (because of the "classroom level" aggregation level) Is this bullet list a reasonable summation of the results?
