# P-value and test power relationship

In this video the author says that

Power of a test is the probability that we will correctly get a p-value below significance level

My question is: when do we know that we correctly obtain a p-value below significance level? To me, it would require knowing apriori that, for example, the two distributions are separable enough. Generally, our p-value $$< \alpha$$ could be also obtained by chance. If I were to estimate the power of a given test, I could draw $$N$$ independent samples from a given distribution and see how many times p-value $$< \alpha$$ and then my estimated power would be $$\frac{\text{number of times p-value }< \alpha}{N}$$, but how do I know how many of these p-values are correctly below $$\alpha$$?

• The power of an experimental design is computed based on a bunch of assumptions. It is not determined empirically. As you point out, that would be impossible. Commented Oct 16, 2023 at 17:36
• @HarveyMotulsky Thank you. So is it justified to estimate the power of a test as a fraction, as I described it above? Even if I don't know which "small" p-values are correct? Commented Oct 16, 2023 at 17:39
• Right. But power is computed for a certain test (with its assumptions) a certain sample size(s), a definition of alpha, an assumption (or decision) about what is the smallest effect you want to be able to detect, and an assumption about how much variability you'll find among the data. Commented Oct 16, 2023 at 18:27
• @ihatepval I agree with Harvey Motulsky comments, but I'm a bit puzzled by the fact that you do not mention the question of sample size and minimal effect size, which are major points in power analysis. I'd suggest to clarify why you want to conduct power analysis in the first place, it might help answering you. Commented Oct 16, 2023 at 18:31

The wording here is perhaps unclear:

Power of a test is the probability that we will correctly get a p-value below significance level

The standard way of talking about this is that power equals $$1 - \beta$$ where $$\beta$$ is the probability of a Type II error, i.e. we fail to reject the null when the null is, in fact, false. So statistical power is calculated conditional on the assumption that the null is false.

Of course, when you implement a statistical test, you don't actually know that the null is false. You also know that there's a $$\alpha$$ probability of rejecting the null even if the null is true.

This is the problem with low statistical power. If you were drawing hypothesis from a hat and you knew that half were true and half were false, and you knew you had 5 percent statistical power and $$\alpha$$ was set to 5 percent, then a significant result would tell you nothing.

• "Of course, when you implement a statistical test, you don't actually know that the null is false". hmm, I'm wondering about that. Isn't it possible to already know that the null is false (e.g. from previous research), and still use statistical tests and apriori power analysis simply to check if a given effect size exists in the population? Commented Oct 17, 2023 at 13:13
• @J-J-J If you know the null is false then the p-value is meaningless, since it is calculated assuming that the null hypothesis is true. You could of choose different (non-zero) effect sizes for your null hypothesis, or consider confidence intervals as the set of not-rejected null hypotheses but that's a different question. Commented Oct 17, 2023 at 13:19
• @GeorgeSavva I guess I have to dig the subject a bit deeper. The paradigmatic case I have in mind is the chi-square test of independence or homogeneity for tables larger than 2x2. Commented Oct 17, 2023 at 13:27