Post-hoc Power analysis to question a non-significant difference? so we did this study about surgical success-rates in two groups (n=84 respectively).
The outcome is binary (success vs. failure).
A priori we expected them not to differ and, indeed, there was no significant difference between the success-rates (63.1% vs. 61.9%).
The reviewer requested a power calculation "to determine the power of this size of study to exclude a difference of any particular magnitude".
Can anybody help me understanding this? I would very much appreciate some guidance!
 A: This reviewer's request is (perhaps characteristically) vague.
The general idea must be to ask what your chances would have
been of detecting a real difference of about $\pm 1.2\%$ in the Success
rates (around $62\%)$ between two groups of size $n=84.$
Various statistical software programs (and, perhaps less reliably,
online calculators) have 'power and sample size' procedures
to answer such questions. Minitab is one of them and its output
is reasonably transparent. Here is output from a recent release of
Minitab that might suffice.
Power and Sample Size 

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.625, α = 0.05

              Sample
Comparison p    Size     Power
        0.63      84  0.050512
        0.65      84  0.063031
        0.70      84  0.176257
        0.80      84  0.711021

The sample size is for each group.


So with $n = 84$ in each group you did not have a high probability
of detecting as, statistically significant, a difference as small as $\pm 2\%, \pm 5\%$ or even $\pm 10\%.$ However, you had a reasonably good
chance of detecting a real difference as large as $20\%$ in either
direction.
The salient issue is how large a difference in Success probabilities would have to be for you to claim an effect that is of practical importance.
Note: Suppose you had $n = 500$ in each group (a much more costly study). For that possibility, the numerical part of the
Minitab output (graph omitted) is as follows:
Power and Sample Size 

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.625, α = 0.05

              Sample
Comparison p    Size     Power
        0.63     500  0.053066
        0.65     500  0.130236
        0.70     500  0.708712
        0.80     500  0.999988

The sample size is for each group.

