I found an awesome R package called
pwr that does all sorts of calculations about sample sizes, power, effect sizes, and so on, and I've been playing.
I have a number of tests that I've run. Now I want to know what kind of power to reject I can get. That's an easy calculation in
pwr. However, if it comes up that I only get $77\%$ when I wanted $80\%$, perhaps I can get $80\%$ power with a minor adjustment to alpha up to $0.06$ instead of the usual $0.05$.
Or maybe I've determined that I want a certain level of power, but I can show that, perhaps I could only get $77\%$ power to reject if I have an effect size of $\delta$, but for some tiny (acceptable) $\epsilon$, I get $80\%$ power with an effect size of $\delta - \epsilon$.
Or maybe I want to play games with the sample size. Perhaps it takes 101 observations to get power up to $80\%$, but if I require $80\%$ power and play games with the sample size to compare with alpha, then for the 100 experiments that I am willing to run, I get $79\%$ power that I consider good enough.
I have a function that plots curves comparing what happens at all of these values: at $\alpha=0.05$, power is $80\%$, at $\alpha=0.06$, power is $82\%$, etc, and ditto for the other pairings (e.g. sample size vs effect size).
I have reservations about doing this, since it feels like p-hacking. At the same time, it feels an awful lot like machine learning using an ROC curve to inform the cutoff threshold for classification.
Is this a legitimate approach? I would (probably) be doing this before I've seen or perhaps even collected the data.