2
$\begingroup$

I found an awesome R package called pwr that does all sorts of calculations about sample sizes, power, effect sizes, and so on, and I've been playing.

I have a number of tests that I've run. Now I want to know what kind of power to reject I can get. That's an easy calculation in pwr. However, if it comes up that I only get $77\%$ when I wanted $80\%$, perhaps I can get $80\%$ power with a minor adjustment to alpha up to $0.06$ instead of the usual $0.05$.

Or maybe I've determined that I want a certain level of power, but I can show that, perhaps I could only get $77\%$ power to reject if I have an effect size of $\delta$, but for some tiny (acceptable) $\epsilon$, I get $80\%$ power with an effect size of $\delta - \epsilon$.

Or maybe I want to play games with the sample size. Perhaps it takes 101 observations to get power up to $80\%$, but if I require $80\%$ power and play games with the sample size to compare with alpha, then for the 100 experiments that I am willing to run, I get $79\%$ power that I consider good enough.

I have a function that plots curves comparing what happens at all of these values: at $\alpha=0.05$, power is $80\%$, at $\alpha=0.06$, power is $82\%$, etc, and ditto for the other pairings (e.g. sample size vs effect size).

I have reservations about doing this, since it feels like p-hacking. At the same time, it feels an awful lot like machine learning using an ROC curve to inform the cutoff threshold for classification.

Is this a legitimate approach? I would (probably) be doing this before I've seen or perhaps even collected the data.

$\endgroup$
4
$\begingroup$

A classic way to proceed is to determine the difference that you wish to be able to detect at a given combination of Type I error and Type II error, and then design a study with a large enough sample size to meet your requirements, given the variability you expect in your measurements.

All of the "playing" you are doing is legitimate in some sense. In particular, "playing games" with sample size is the classic application of power calculations for study design. You should nevertheless be aware of the following.

First, the standard frequentist requirement for "statistical significance" is p < 0.05, so if you plan on publishing the results of your study it would be wise to stick with $\alpha = 0.05$.

Second, the power represents your willingness to miss a truly "significant" (in the sense noted above) result. If you are willing to risk a 21% or 23% chance of such a miss then you can go ahead with 79% or 77% power. A funding agency or a supervisor, however, might not be willing to go so low. 80% power is commonly used; you might want to go even higher.

Third, what is most important (if perhaps most difficult) is having a good handle on the difference you would like to detect and the variability in the measurements. If you have made an unreasonable choice for the difference you're trying to detect, then no playing with power calculations will help. If you don't have a reliable estimate of measurement variability, the power calculations will be correspondingly unreliable. Focus on those matters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.