Imagine a researcher conducted an experiment and tested an hypothesis using a two independent samples approach. It was the first experiment of this kind, so the researcher had no idea about a possible effect size - therefore estimated the sample size based on an a priori analysis, based on an expected mean difference, divided by a "guessed standard deviation".

But it turns out that the achieved sample size wasn't perfectly the planned, and also the standard deviations were really different (smaller) from the initial guess. To ensure the conducted study had enough power, the researcher used the initially predicted mean differences, divided by the actually observed standard deviation as effect sizes, and the obtained sample sizes. It made possible to confirm the study had enough power and the rejection of the hypotheses were not a failure to reject a false null hypothesis.

The problem is: how to report this analysis? The name "post hoc" would be surely misleading, and also it's not exactly a post hoc analysis, as the effect size is based on a a priori expected mean differences. Yet it's not an a priori power analysis too.

  • 1
    $\begingroup$ The study failed to reject the null, so it didn't have enough power. That is about all there is to it. This is all assuming the "true" mean difference is not exactly zero, which is very likely. See "design analysis" as an alternative andrewgelman.com/2017/03/03/… $\endgroup$ Commented Nov 9, 2018 at 9:39
  • $\begingroup$ @HeteroskedasticJim I liked the proposed approach, it looks very promising, for sure. Also, I agree that (obviously) if the study failed to reject the null it doesn't have enough power - under the standard post hoc analysis, for obvious reasons. Yet, the way I have calculated it - keeping the a priori expected mean difference, just "updating" standard deviation - concluded it rejected with enough power, because I took practical significance rather than statistical significance. $\endgroup$
    – Ágatha
    Commented Nov 10, 2018 at 17:21


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.