# ROC-style curves for calculating sample size, power, alpha, and effect size

I found an awesome R package called pwr that does all sorts of calculations about sample sizes, power, effect sizes, and so on, and I've been playing.

I have a number of tests that I've run. Now I want to know what kind of power to reject I can get. That's an easy calculation in pwr. However, if it comes up that I only get $$77\%$$ when I wanted $$80\%$$, perhaps I can get $$80\%$$ power with a minor adjustment to alpha up to $$0.06$$ instead of the usual $$0.05$$.

Or maybe I've determined that I want a certain level of power, but I can show that, perhaps I could only get $$77\%$$ power to reject if I have an effect size of $$\delta$$, but for some tiny (acceptable) $$\epsilon$$, I get $$80\%$$ power with an effect size of $$\delta - \epsilon$$.

Or maybe I want to play games with the sample size. Perhaps it takes 101 observations to get power up to $$80\%$$, but if I require $$80\%$$ power and play games with the sample size to compare with alpha, then for the 100 experiments that I am willing to run, I get $$79\%$$ power that I consider good enough.

I have a function that plots curves comparing what happens at all of these values: at $$\alpha=0.05$$, power is $$80\%$$, at $$\alpha=0.06$$, power is $$82\%$$, etc, and ditto for the other pairings (e.g. sample size vs effect size).

I have reservations about doing this, since it feels like p-hacking. At the same time, it feels an awful lot like machine learning using an ROC curve to inform the cutoff threshold for classification.

Is this a legitimate approach? I would (probably) be doing this before I've seen or perhaps even collected the data.

First, the standard frequentist requirement for "statistical significance" is p < 0.05, so if you plan on publishing the results of your study it would be wise to stick with $$\alpha = 0.05$$.