How to compare powers of two tests based on their p-values?

Question

Suppose we are testing null $H$ versus $K$. Here $H$ and $K$ are viewed as two disjoint sets of distributions of sample $X$

Given a sample $X$, one test rule rejects null if and only if $T_1(X) \geq c_1$, and the other $T_2(X) \geq c_2$.

I was wondering how to compare the powers of the two tests at a sample distribution $F \in K$, based on their p-values?

One way I saw from an example (without explanation) is to first fix a level of significance $\alpha$, and then calculate the probability of each test's p-value no greater than $\alpha$, when the sample distribution is $F \in K$, i.e., $$ \mathrm P_{X \sim F} \{[ \sup_{F' \in H} \mathrm P_{X' \sim F'} (T(X') \geq T(X))] \leq \alpha \} $$

The bigger the value is, the more power the test has at $F \in K$, but I was wondering why this is true?

Does it depend on $\alpha$?

Answer 1

Under a simple null hypothesis, the sampling distribution of the p-values is uniform. Under a composite null, it is stochastically larger, and under the alternative, it is stochastically smaller. How small? Depends on the alternative. Informally- If it is well separated from the null distribution, it will be much smaller. If the alternative data distribution is "epsilon close" to the null, the p-value CDF will be "epsilon close" to uniform.

If you can compute the CDF of the p-value, you can invert it to get the parameter's value, and the compute the power as you would typically do.