4
$\begingroup$

I am writing a paper for a journal. I have been asked to calculate the statistical power of my study. I have zero idea about how to do that. I am an engineer and never cared about this kind of analysis, but others do. I was told to use GPower software for this, however, I don't know where to begin.

I have a study with $68$ records, files, that I have used for a test. From each of them I have $3$ different events to detect. I have a total for $521$ for event $A$ and $6318$ for event $B$. The number of events $A$ in each record is about $5 ± 1$, and for the events $B$, $80 ± 50$.

So now, how do I compute the statistical power? Where do I begin?

$\endgroup$
11
  • 8
    $\begingroup$ To calculate statistical power you need to state a null hypothesis that you want to test and state a method that you will use to test it. $\endgroup$
    – Peter Flom
    Commented Mar 3 at 21:24
  • 4
    $\begingroup$ Just to clarify, are you meaning that you have your test results already and that the journal wants you to do a power analysis post-testing? $\endgroup$ Commented Mar 3 at 21:49
  • 3
    $\begingroup$ Correct, they are asking me for the post analysis. $\endgroup$
    – GGChe
    Commented Mar 3 at 22:15
  • 8
    $\begingroup$ If they want a calculation of post hoc power, you'd be doing your area of study a favor by explaining that it should not be used. Many posts and comments on site discuss the problems. e.g. see the Gelman reference in mkt's answer and the the Honig & Helsey reference in dipetkov's answer here, and and the discussion here. Searches should yield more $\endgroup$
    – Glen_b
    Commented Mar 3 at 22:27
  • 1
    $\begingroup$ The Wikipedia entry "Power of a Test" has a useful summary and references for why post-hoc power analyses are a bad idea. $\endgroup$ Commented Mar 4 at 3:08

1 Answer 1

12
$\begingroup$

Answer

As others have already noted in the comments, you should never do a post-hoc power analysis. All three of the references below can be easily cited as defenses for yourself if the reviewer doesn't accept this.

I will note however that a priori power is indeed useful if you do not have data already collected, so there is a meaningful difference between the two.

References

  • Dziak, J. J., Dierker, L. C., & Abar, B. (2020). The interpretation of statistical power after the data have been gathered. Current Psychology, 39(3), 870–877. https://doi.org/10.1007/s12144-018-0018-1
  • Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1), 33267.
  • Zhang, Y., Hedo, R., Rivera, A., Rull, R., Richardson, S., & Tu, X. M. (2019). Post hoc power analysis: Is it an informative and meaningful analysis? General Psychiatry, 32(4), e100069. https://doi.org/10.1136/gpsych-2019-100069
$\endgroup$
12
  • 3
    $\begingroup$ +1. I would very much add Hoenig & Heisey, "The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis" (The American Statistician, 2001), simply because it is in a "general" statistics journal, whereas the above references are all from psychology/psychiatry and may thus not be completely convincing to engineer reviewers. $\endgroup$ Commented Mar 4 at 6:57
  • 1
    $\begingroup$ Thanks Ill add that to my list. Good to have something more broad anyway. $\endgroup$ Commented Mar 4 at 7:40
  • 2
    $\begingroup$ Thanks everyone for the help. This is a very nice information. More literature for my backlog. Always good to have and not very old. Thanks ! $\endgroup$
    – GGChe
    Commented Mar 4 at 18:45
  • $\begingroup$ A calculation for the power of a study is a useful information about the precision of a study similar to a confidence interval. This can still be computed a posteriori. A two one sided t-test is a bit similar to this. You do not only tell the p-value for the hypothesis of equivalence, but also the p-value for the hypothesis of difference. $\endgroup$ Commented Mar 5 at 12:33
  • 1
    $\begingroup$ @SextusEmpiricus When you say a confidence interval 'might' also work, what is the question that you are seeking to answer with a post-test power analysis? Before we can address how to do the power calculation, it is still unclear to me why we should. What extra do these power analyses provide? (In contrast, the reasons for testing and for confidence intervals are clear.) $\endgroup$ Commented Mar 5 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.