Interpretation of power calculations

Question

We got the following comment from an associate editor who asked for ex-post power calculations. We used the treatment difference and sample size from our study and reported the power of detecting the effect of the observed point estimate at a 5% significance level. The AE was not happy and wrote:

It appears that you used your existing data. However, power calculations are something one does (or should do) ex ante, i.e., when the treatments averages are unknown. it continued "More to the point, the idea of the power calculations is to test for the probability of type I and type II errors. This, by definition, means that the sample mean and standard deviation are likely to be different than those in the population. "

If I am not mistaken, I can not jointly test (determine, I think is what s/he meant), as one has to fix at least one of the errors. I am really not sure what to make out of this comment.

Answer 1

The reason for doing power calculations is to determine ahead of time how large a sample ($n$) would be needed to detect a particular difference, which we can call effect size. From Power Analysis, Statistical Significance, & Effect Size, one reads (at the end of it)

"How do I estimate effect size for calculating power?

Because effect size can only be calculated [sic, estimated] after you collect data from program participants, you will have to use an estimate for the power analysis. Common practice is to use a value of 0.5 as it indicates a moderate to large difference."

After collecting data, one can calculate what effect size is (should have been) detectable. This becomes important when no significant effect is detected in the experiment. Because that information is missing in the question, i.e., Was the result of the difference A) significant or B) not significant? If not significant, then what difference would have been detected becomes an issue. For example, suppose that the difference between two different measurement methods of renal clearance is 0 where the average value for each is 454 ml/min. Does this mean that those methods are the same? Hardly, and this is a common error that experimenters make, to take a not significant result and claim significance. Here is an example from the literature,

"To justify their claim of 100% renal clearance at 48 h, Pentikainen et al set total clearance equal to renal clearance. However, they did not substantiate that equality numerically. Although their average value of total clearance was only 5.7 ml/min greater than renal clearance, the 95% confidence interval for this difference is from −173 to 184 ml/min, such that their insignificant difference in clearance by Student's $t$ test did not exclude a huge potential range of clinically significant difference."

So does this answer the question, or, what is the question, anyway?

Answer 2

According to one 'authoritative source' I just consulted on the Internet the average height of US women is $\mu_0 = 5.35$ ft. Suppose I believe that women students at my university are taller than that. So I want to test $H_0: \mu = 5.35$ against $H_a: \mu > 5.35$ at the 5% level of significance.

I plan to sample $n$ women at random from my university and I would like to be 'pretty sure' of rejecting $H_0$ if women at my university average 'substantially' more than 5.35' in height.

Various sources say that the standard deviation of US women's heights is about 3" or about 0.25'. That would mean almost all US women are between $5.35 \pm 0.75$ feet, which seems roughly reasonable. If 'pretty sure' means a power of 95% and 'substantially` means more than 0.1' (a little more than an inch), then I have the information necessary to estimate the number $n$ of subjects I need in my sample.

Here is output from Minitab's 'power and sample size' procedure. (Many statistical software programs will perform similar computations.)

Power and Sample Size 

1-Sample t Test

Testing mean = null (versus > null)
Calculating power for mean = null + difference
α = 0.05  Assumed standard deviation = 0.35


            Sample  Target
Difference    Size   Power  Actual Power
       0.1     134    0.95      0.950080

So I will have to measure heights of about 134 women at my university to meet my requirements for the design of the study.

Of course, as several people have commented, the time for a power and sample size computation is before you start taking data. What the AE may be asking you to do is to make it seem as if you had some reason to believe in advance that your sample size would be adequate.

You cannot ethically claim to have done a power computation in advance. However:

• You might be able to cite other studies making measurements similar to yours to see the standard deviations of your measurements; or maybe you did a few trial runs before the experiment and have your own estimate for $\sigma.$

• You must have had some idea how big a difference would be of sufficient practical importance to get your work published.

• Also, it would have been reasonable to aspire to a $90\%$ or $95\%$ chance of detecting a difference of important size with whatever number of subjects you used.

Without making false claims, you could talk generically about power and sample size as outlined just above. Maybe you are astonished to have found any effect at all or maybe you had wished to find a larger effect, but it might be helpful to readers of your paper to have some context for what must (should?) have been in your mind when embarking on the experiment.

Moreover, maybe the AE is hoping your next submission shows evidence of a cogent power and sample size study at the beginning. Do that, make notes, and save relevant computer printouts.