P-value and test power relationship

Question

In this video the author says that

Power of a test is the probability that we will correctly get a p-value below significance level

My question is: when do we know that we correctly obtain a p-value below significance level? To me, it would require knowing apriori that, for example, the two distributions are separable enough. Generally, our p-value $< \alpha$ could be also obtained by chance. If I were to estimate the power of a given test, I could draw $N$ independent samples from a given distribution and see how many times p-value $< \alpha$ and then my estimated power would be $\frac{\text{number of times p-value }< \alpha}{N}$, but how do I know how many of these p-values are correctly below $\alpha$?

Answer 1

The wording here is perhaps unclear:

Power of a test is the probability that we will correctly get a p-value below significance level

The standard way of talking about this is that power equals $1 - \beta$ where $\beta$ is the probability of a Type II error, i.e. we fail to reject the null when the null is, in fact, false. So statistical power is calculated conditional on the assumption that the null is false.

Of course, when you implement a statistical test, you don't actually know that the null is false. You also know that there's a $\alpha$ probability of rejecting the null even if the null is true.

This is the problem with low statistical power. If you were drawing hypothesis from a hat and you knew that half were true and half were false, and you knew you had 5 percent statistical power and $\alpha$ was set to 5 percent, then a significant result would tell you nothing.