# How is power of an experiment practically useful when p-value is low?

Let's say my null hypothesis distribution looks like this

The red lines form the distribution and I've marked black lines corresponding to a significance level of 0.05.

Now I run an experiment and get a uniform distribution that looks like the larger green rectangular region below.

The interesting thing is, the mean of the experiment's distribution is quite high and outside the "normal" defined by the significance value of 0.05. So the p-value of turns out to be < 0.05 and the experiment is thus statistically significant.

But since the distribution is so wide, its power (shaded green region) is too low (55%).

Now I'm confused how to move forward here. The p value is low enough that we can reject the null hypothesis. Since we reject the null hypothesis, a type II error is out of the question. Type II errors happen when you fail to reject the null hypothesis when the null hypothesis is wrong. Here we already have confidence that to reject the null hypothesis from the low p-value.

What am I missing here? Clearly the alternative hypothesis is not a good one from how wide the distribution looks. What would be a methodical way of going about deciding to choose the alternative hypothesis presented here?

• When the p-value is low you will proceed to act as if there is a real effect. In that case, why would you care about the power?
– whuber
Commented Jul 25, 2022 at 14:06
• @whuber exactly, so is it correct to ignore power in this scenario? My understanding is, the only scenario when power is useful would be when you're not confident from looking at the p-value, so you're considering going with the null hypothesis. So you check power and if the power is low, you consider increasing the sample size or increasing alpha before abandoning the alternative hypothesis. Does that sound right? Commented Jul 25, 2022 at 14:33
• Those are all reasonable considerations.
– whuber
Commented Jul 25, 2022 at 15:41
• A low p-value from an underpowered experiment is more likely to be a false positive. Illustrative example: no data, p-value generated as uniform on 0 - 1. p <= 0.05 also most certainly does not mean that a type I error is out of the question... Commented Jul 25, 2022 at 15:45
• @Björn good point, I think that answers the question. So for conclusive statistical significance, we need both low p value and high enough power. I wonder why this is not stressed in most places. Even on Wikipedia, they allude only to p-value < alpha as the requirement. Commented Jul 25, 2022 at 16:43

In this theoretical example, the $$p$$-value wouldn't correspond to the mean of the actual distribution of the test statistic (the green one in your schematic). Rather, the $$p$$-value would come from a single realization from it.

Another point of confusion is exactly what these "distributions" represent. There are two concepts here. The distribution of the data and the distribution of the test statistic. In null hypothesis significance testing, we only care about the latter.

As you know data are summarized with a statistic, such as a mean or a median. If experiments of a fixed size (say N=30) were replicated according to a probability model (say 1,000s of times), and the test statistic were calculated for each replication, you wind up with the distribution of the test statistic.

In your example, I have assumed that the triangular and uniform densities correspond to the distribution of the test statistic. Only in that case does it make sense to compute critical values, and percentiles of the distribution as power. When it comes to actual data analysis, we rarely know the actual distribution of the test statistic either under the null or the alternative. But during the planning and sample size calculation phase of the study, we make the minimal number of assumptions to create exactly the graphical depictions you've shown here.

• Isn't this the usual way of running A/B tests? You want to understand if a new Ad recommendation model's click through rate (CTR) is better, so you collect CTRs for the new model by sending it some user traffic and take the mean of that CTR distribution as the model's CTR. Then p-value is measured as "how rare was this CTR in the null hypothesis distribution?". Commented Jul 25, 2022 at 12:54
• @tensorsk a response isn't fit for a comment so I expanded the answer. Commented Jul 25, 2022 at 13:08
• Hey, thanks a lot for the answer. I studied this over the week and it's making more sense. So finding power of an experiment is done in the preparatory phase. After we find the power, we believe that future single realizations made from the hypothesis will have probability=power of resulting in a p-value < alpha. Commented Aug 1, 2022 at 17:57
• @tensorsk Yes, power calculation is a preliminary (never post-hoc) procedure. Once we understand the distribution of the test statistic under the null hypothesis, we can run our experiment. A well powered test has the distribution of the test statistic under the alternative tending toward the tails of the null distribution, beyond whatever the critical value is. Once the experiment is run, we calculate the quantile of that value according to the null. A very low probability, i.e. one less than alpha, is the decision rule to reject the null. Commented Aug 1, 2022 at 18:48
• > "Once we understand the distribution of the test statistic under the null hypothesis, we can run our experiment." You mean "once we understand the distribution of the test statistic under the null and alternative hypotheses" right? In order to find power, we need both distributions. So we would have to run both the control and alternative hypothesis experiments and use something like bootstrapping (or run them many times) to plot those distributions? Commented Aug 2, 2022 at 1:30