0
$\begingroup$

I am running a simulation where I know the null hypothesis is true.

The power of a hypothesis test is the probability of making the correct decision if the alternative hypothesis is true. That is, the power of a hypothesis test is the probability of rejecting the null hypothesis H0 when the alternative hypothesis HA is the hypothesis that is true.

Is there any terminology for the probability of accepting the null hypothesis when it is true?

IE, suppose I have rejection rate of .05 for a particular test where I know the null hypothesis is true. What does this say about the power of this test, if anything?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ If there is a 5% probability of falsely rejecting a true null hypothesis then there is a 95% probability of correctly failing to reject it. (Power doesn't really mean anything in the context where the null hypothesis is true, just as the false positive error rate is not relevant when the null is false.) Note that there are more ways to make false inferences than just false positive and false negative errors. Look up type M and type S errors, for example. $\endgroup$
    – Michael Lew
    Commented Dec 7, 2016 at 2:43
  • $\begingroup$ @Michael I think your comment would make a suitable answer. $\endgroup$
    – Glen_b
    Commented Dec 7, 2016 at 13:20

1 Answer 1

Reset to default
1
$\begingroup$

If there is a 5% probability of falsely rejecting a true null hypothesis then there is a 95% probability of correctly failing to reject it when it is false.

Power doesn't really mean anything in the context where the null hypothesis is true, just as the false positive error rate is not relevant when the null is false.

While textbooks and people with a frequentist perspective often make it sound like the only relevant errors are type I and type II errors (false positive and false negative errors), there are many ways to make erroneous inferences. In particular it is useful to recognise that you can be mistaken about the magnitude of an effect (type M error) and even about the direction or sign of the effect (type S errors). The latter errors are only common among low powered studies but, because low powered studies are more common that one might like, they are a serious problem.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.