1
$\begingroup$

In psychology papers that do prospective power analyses (example), one often notices the convention of assuming β values (false negative rates) of 0.20, and power levels of 0.80. In other words (if I understand correctly), we are happy with a mere 80% chance of detecting true effects, and are happy to accept a whopping 20% chance of failing to detect such true effects.

Both values seem unacceptably low/high (respectively), which I was first tempted to cynically associate with claims about psychology not being a real science, namely on the low cost placed on finding false effects or on NOT finding true effects.

On second thought, however, it does stand to reason that the more you require of one criterion (e.g. low β), the more you necessarily have to compromise on the other (e.g. make do with a small power). In other words, that there be a trade-off between them.

I have trouble understanding however whether it is a coincidence that these two measures - in this case - add up to 1, and whether this has to be the case that β+power=1.

Improve this question
$\endgroup$
1
$\begingroup$

To be honest, I am not a big fan of the 2x2-table that is typically presented to explain the relationship between $\alpha$ and $\beta$ (or power, respectively) because it looks like the total sum of probabilities should be $100\%$ and does not acknowledge the total separation of the true status of $H_0$. So I'd rather present it the following way:

For a statistical test, you can imagine two scenarios: either the null hypothesis, which you are trying to reject, is actually true or is not true.

If $H_0$ is true:

  • if you reject, you're doing a type I error (probability for this is $\alpha$)
  • if you do not reject, you're doing fine (probability $1-\alpha$)

If $H_0$ is not true:

  • if you do not reject, you're doing a type II error (probability $\beta$)
  • if you reject, you're doing fine (probability for this is $1-\beta$, which is called the power)

Here, you see that $\alpha$ and $\beta$ are not directly connected to each other. They simply describe the probability of doing the false decision in one of the two scenarios. And you see that - if the null hypothesis is not true - you have exactly two options: to reject (correct decision, probability $1-\beta$ which is the power) or not to reject (wrong decision, probability $\beta$).

I agree with you that $\beta=20\%$ for many settings is way to high, since it corresponds to $20\%$ chance of the study to fail. If I were a PhD-student and the study that I will conduct and analyze for the next three years, I would definitely not want take the odds to go with only $80\%$ power ...

Improve this answer
$\endgroup$
7
  • $\begingroup$ OK, so to clear up my confusion in a nutshell: power is 1-β, not 1-α. This would also mean the answer to the question of my OP is "yes". RIght? $\endgroup$ – z8080 Feb 6 '19 at 16:05
  • $\begingroup$ Indeed. Per definition, in hypothesis testing: β+power=1, or, respectively, power=1-β. $\endgroup$ – LuckyPal Feb 6 '19 at 16:11
  • $\begingroup$ And α only applies to the case, when the null hypothesis is true. Then, there is no β and no power. Those two terms only exist when the alternative hypothesis is true (and then there is no α). $\endgroup$ – LuckyPal Feb 6 '19 at 16:12
  • 1
    $\begingroup$ @z8080 actually there is no true positive/negative rate in hypothesis testing. This is because there is no probability that H0 is true - it simply is true or not true. This is concept I also found difficult to grasp but it is crucial for hypothesis testing. There is only a probability for the case of wrong/correct decision if H0 is true, and another probability for the case of wrong/correct decision if H0 is not true. $\endgroup$ – LuckyPal Feb 7 '19 at 9:56
  • 1
    $\begingroup$ Thus, if the null hypothesis is true, the probability of correct decision is 1-α. The concept of true negative rate and true positive rate does not make sense here, because if the null hypothesis is true, then per definition there are no true positives and no false negatives. There are only true negatives (1-α) and false positives (α). $\endgroup$ – LuckyPal Feb 7 '19 at 9:58
2
$\begingroup$

This is no coincidence, this is exactly how these measures are defined.

enter image description here

Edit:

You are mixing notations and tables, the table above is a Table of error types, which is used in statistics and inference / hypothesis testing. True positive rate (TPR) aka Recall aka Sensitivity comes from a confusion matrix based on predictions.

enter image description here

$TPR = TP/(TP+FN)$

Improve this answer
$\endgroup$
6
  • $\begingroup$ I thought 1-β is the true positive rate, aka sensitivity. Surely this is not the same as power, otherwise we would not have different terms?.. $\endgroup$ – z8080 Feb 6 '19 at 13:57
  • $\begingroup$ true positive rate refers to the probability that the alternative hypothesis is correct. In statistical test theory, we do usually assume that a hypothesis is true or not - so there is no probability that the hypothesis is correct. We can only check, under the assumption that the alternative hypothesis is correct, what is the probability of rejecting the null? $\endgroup$ – LuckyPal Feb 6 '19 at 14:17
  • $\begingroup$ @LuckyPal I don't know what you are talking about, Sensitivity and specificity are statistical measures of the performance of a binary classification test. I don't see their connection to hypothesis testing. $\endgroup$ – user2974951 Feb 6 '19 at 14:51
  • $\begingroup$ @user2974951 it's a very frequent mistake to apply the concept of specificity to hypothesis testing and I think this is exactly what caused the confusion of the OP, so I addressed this issue and explained what a true positive rate would mean in the context of hypothesis testing. Actually, in a Bayesian way to think about hypothesis testing you can think of hypothesis testing as a binary classification problem. $\endgroup$ – LuckyPal Feb 6 '19 at 15:33
  • $\begingroup$ @user2974951 a very good description of what I mean is given here: Nuzzo, Regina. "Scientific method: statistical errors." Nature News 506.7487 (2014): 150. $\endgroup$ – LuckyPal Feb 6 '19 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.