Low power for an experiment?

Question

I am running an A/B experiment and trying to do a regression analysis on the test and control mean. I do not have enough samples to run the experiment at 80% power. What does it mean for the treatment effect that would come out from the regression if I do proceed with the underpowered test ? What can I say regarding the output beta value for coefficient of regression ?

Answer 1

Low power at what effect size? If the true effect size is larger than the value that you plug into the calculation of power then you would have more power than you think. Conversely, if the true effect is smaller or the variance is larger than the values that you plugged into the analysis then you will have less power.

Power is best viewed as a function rather than a singular value. See section 2.5 of this chapter: https://link.springer.com/chapter/10.1007/164_2019_286 and bear in mind that low power increases the chances that a 'significant' result is a product of the "significance filter exaggeration machine" explained in section 3.1. That means that a significant result from a low-powered experiment is more likely to have an exaggerated effect size than an equivalent outcome from a well-powered experiment.