1
$\begingroup$

Is it incorrect to say that $\text{Power} = P(\text{reject} H_0)$?

I've seen in my textbook as well as other sources power is always defined as the conditional probability $P(\text{reject } H_0 : H_a \text{ is true})$

But I also see in hypothesis testing you are to regard the probability of the null hypothesis as being either $1$ or $0$, since it is treated as a constant when computing the test statistic. So in Baye's theorem, $P(\text{reject } H_0 : H_a \text{ is true})=\frac{P(H_a \text{ is true}:\text{reject } H_0)P(\text{reject } H_0)}{P(H_a \text{ is true})}$

In which case if $P(H_a\text{ is true}) =1$, then $P(H_a\text{ is true} : \text{ reject } H_0)=1$

So $\text{Power} = P(\text{ reject } H_0)$

But if $H_a$ is not true, then $P(H_a\text{ is true}) =0$ I'm not sure what the power means in this case. The other part I find confusing is if my test statistic is coming from a continuous distribution, then don't I already know that $P(H_0 \text{ is true })=0$ and wouldn't then $P(H_a \text{ is true})=1$, so wouldn't it always be the case that $\text{Power}= P(\text{ reject } H_0)$?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

As Michael points out above, power is the probability of rejecting the null when it is false. Now when you're empirically simulating the power of a particular test to detect a particular effect, you will know that the null is false and in this case the empirical power is (as you suggest above) simply the the probability that the null is rejected. For example, if you want to know the power of a particular test to detect a difference of $d$ between $x$ and $y$, you draw 10,000 samples from $x$ and $y$ and for each sample test whether you can reject the null. Here the power is simply the proportion of samples where you rejected the null. This is rather a special case though since you know that the null is false, e.g. you are drawing from $x$ and $y$ which differ according to $d$.

In most cases, say when you're planning an experiment, you don't know if the the null is false. Sure, you know that the null is either true (1) or false (0) but knowing this doesn't give us the prior probability of 1 (or 0), e.g. when we flip a coin we know that it will either be heads (1) or tails (0), but that doesn't give us the probability of 1 (or 0). Now the tricky part here is that if we knew whether the null was true or false, we wouldn't need an experiment and thus wouldn't need a power calculation.

Sometimes we might be able to construct an estimate for the prior probability that the null is true, for example, in testing relationships between genetic variants and disease. In most cases where we need to calculate power, though, we won't have a very good estimate for the prior probability and so won't be able to use Bayes Theorem.

My answer here and the links therein might help clarify this for you: Is there such a thing as a p-value that incorporates a priori power estimations?

Improve this answer
$\endgroup$
3
  • $\begingroup$ I guess I'm still confused because it seems like when computing a test statistic, that if the null hypothesis is being treated as a constant, then it doesn't follow a distribution. So how would it make sense for it to have a probability? $\endgroup$
    – AColoredReptile
    Commented Mar 13, 2022 at 22:11
  • $\begingroup$ Yes, you calculate the test statistic under the assumption of the null hypothesis but that doesn't mean the null hypothesis doesn't have a prior probability. Did you look at the papers linked in the response I linked? $\endgroup$
    – num_39
    Commented Mar 14, 2022 at 5:01
  • $\begingroup$ Maybe it helps to put it like this. You calculate the test statistics under the null, i.e. if the null is true, how likely am I to get this test statistic, but you don't know that the null is true. If you did, you wouldn't need a test. Typically, you don't have enough information to say much about the probability that it is true but sometimes you might. Think about betting markets for replications of RCTs. $\endgroup$
    – num_39
    Commented Mar 14, 2022 at 5:49
0
$\begingroup$

Yes, it is incorrect to say that power is the probability of rejecting the null. It is the probability of rejecting it when it is false.

Your statement about the probability of the null (being true) is wrong. The test statistic is computed from the data and does not require any truth value to be ascribed to the null hypothesis.

Improve this answer
$\endgroup$
4
  • $\begingroup$ For a t test though isn't $t=\frac{\bar x - \mu_0}{s/ \sqrt{n}}$ isn't that $\mu_0$ the null hypothesis, say that $\mu_0=0$? $\endgroup$
    – AColoredReptile
    Commented Mar 12, 2022 at 21:26
  • $\begingroup$ @AColoredReptile Good point! As someone who typically writes the t formula like that I should have known better! $\endgroup$
    – Michael Lew
    Commented Mar 12, 2022 at 21:29
  • $\begingroup$ So is it not true that the probability of the null is $1$ or $0$? My understanding was that the reason for instance the t test statistic follows a t distribution is because $\mu_0$ is not a random variable. And constants have trivial probabilities of $1$ or $0$. $\endgroup$
    – AColoredReptile
    Commented Mar 12, 2022 at 21:36
  • $\begingroup$ In the sense that the probability of something that can be either true or false is zero or 1, then yes. However, consider a set of simulations which are set up to explore the test in question. Some of the runs will have true nulls and others false. Even if the probability of the null being true in each individual run can be called 0 or 1, the aggregate probability of nulls being true will have a fractional value. Now think of the probability of nulls being true in real analyses of actual data. Are they most usefully described as 0 or 1, or as an unknown fraction? $\endgroup$
    – Michael Lew
    Commented Mar 13, 2022 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.