I need a little help or reassurance concerning how to explain power "to power", i.e. to decision makers that are not well versed in statistics.

The problem is this: I have done three empirical studies on a treatment and the effect on an outcome. The treatment ranges in value from small to fairly large. There is no natural way to pool the treatments, since they differ in terms of target population, period of study and "internal treatment logic", i.e. what is done to the treated. The target outcome is the same for all studies. You can think of it as giving some classrooms extra math lessons,

Study 1 - 40 more hours Study 2 - 20 more hours Study 3 - 10 more hours All other classrooms - 0 more hours

And then measuring the improvement in a test score for math skills across the whole school.

For all studies, I use a matching approach (coarsened exact matching) on covariates pre-treatment, and then a weighted multiple linear regression (weights are proportions of treated to untreated in matching strata). I find effect sizes of between 5 and 20 per cent compared with an untreated population (pupils in all other classrooms). (

Study 1 is sufficiently powered - large number of treated (large class, no "new" pupils), good matching - and those give me type-I significant estimates of about 15 per cent. This could be expected and 15 percent is a reasonable result, since this is the "large" treatment.

The two other studies are more problematic. Study 2 has few treated (but a reasonable matching and number of controls), but I estimate an effect size of almost 20 per cent, not statistically significant (p=0.14)

Study 3 has an effect size of around 8 per cent, not significant (p=0.39) and a large variance in the outcome

Both studies 2 and 3 are underpowered, using an guesstimated effect size of maybe 10 percent for study 2 and maybe 5 percent for study 3. at $\alpha=0.9$ and $\beta=0.7$ (don't ask for a rationale for these numbers, they just seem reasonable... )

Now I need to summarize this over "increased math hours program".

I would like to say is that study 1 is "generalizable", i.e. I can use study to guide expectation and expect the outcome to be positive. I cannot say that for study 2 and 3 - in those cases the results are valid for the pupils studied. I feel like an undergraduate statistics student again, but is this an accurate description of the inference?

  • I expect a positive effect on math score for all untreated classrooms with 40 extra math hours
  • I would not have a reasonable guess as to the effect of giving less than 40 hours (20 or 10 hours). A bigger study with more pupils is needed to capture smaller expected effect sizes.
  • All programs improve average math scores for the pupils in the study

I need some reassurance, because some stakeholders (teachers?) interpret insignificant as "no effect whatsoever" and I am beginning to believe them (because of the "classroom level" aggregation level) Is this bullet list a reasonable summation of the results?

  • 1
    $\begingroup$ I suggest you discuss the significance tests, CIs, etc. but leave power out of the discussion. Power has no relevance to data that have already been collected. It is only useful in planning studies to ensure the sample size is adequate for the study goals. $\endgroup$
    – Russ Lenth
    Sep 28, 2014 at 20:16
  • $\begingroup$ I kind of get where you are coming from... But I want to suggest something from these results. A suggestion to either (a) avoid classroom 3 evaluations or (b) push for better designed classroom 3 studies. I do not want to be in a discussion if the estimated effect is "real" or not. Explaining that the minimum detectable effect in the current data is larger by a magnitude (of 4!) would kind of support my suggestion. I hope. $\endgroup$ Sep 29, 2014 at 6:23
  • 2
    $\begingroup$ I agree with that. There's nothing wrong with looking at the estimated effect. In fact, it's kinda wrong not to! But you can make that point without mentioning power: "The effect we observed is huge, but not significant. The sample size is too small to detect such an effect due to the large amount of variation in the data." $\endgroup$
    – Russ Lenth
    Sep 29, 2014 at 21:46


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.