1
$\begingroup$

I'm computing the power of a statistical test using Monte Carlo simulation. My test statistic is asymptotically $\chi^2$ under the null hypothesis. When I am computing the power for a given iteration of my simulation, I use the $\chi^2$ distribution to see whether I should reject in that iteration even though the test statistic may not be distributed $\chi^2$ under the alternative. Why do we do this?

Is this the entire point of the test in some sense? Are we saying "Oh, this is how I would behave (accept/reject) if the null were true (i.e. the test was asymptotically $\chi^2$), so let's see how often we reject the null when we think it's true but in fact it's false"?

$\endgroup$

1 Answer 1

1
$\begingroup$

The null distribution is used in choosing your rejection rule, since that determines your type I error rate (or an upper bound on it).

Once you have a rejection rule, you're using it to decide whether to reject $H_0$, whichever hypothesis applies (you're always using the same rejection rule, since you don't know which hypothesis applies; that's the point of the test, after all).

Power is the probability of rejecting $H_0$ using that rejection rule (determined under the null), for some particular non-null effect size.

Are we saying "Oh, this is how I would behave (accept/reject) if the null were true (i.e. the test was asymptotically χ2), so let's see how often we reject the null when we think it's true but in fact it's false"?

Not at all. We don't "think it's true". We don't know.

We determine the rejection rule (e.g. "reject when the test statistic $\geq c$") using the distribution under $H_0$, because that allows us to control type I error. We will then use that rejection rule in the test where we don't know which hypothesis is true.

We do want to know how good the test is at rejecting $H_0$ in a situations where it's false. Naturally we are still using that rule ("reject when the test statistic $\geq c$" say) in that situation.

$\endgroup$
1
  • $\begingroup$ I think this makes sense, thank you for the clarity! $\endgroup$
    – student_t
    Sep 23, 2022 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.