Does the minimum of the power of a test always have to equal $\alpha$ the significance level? I'm comparing the power of 2 tests, one always has its minimum at $\alpha$, while the other have its minimum somtimes below $\alpha$, other times it's above $\alpha$, it depends on the sample size. Is this behavior normal? what does this say about the second test?
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
Tests will under the null hypothesis or with an effect that is very small (i.e. not very far away from the null hypothesis) typically have a power close to their significance level. Once you move into the interior of parameter set that constitutes the null hypothesis (if there is such a thing), their "power" will normally be lower than that.
Reasons for deviations from this include
- The test is conservative and does not fully exhaust the significance level. This for example is known to be the case for Fisher's exact test and other exact methods for small sample sizes - with a small discrete outcome space, exact methods exact methods keep the significance level $\leq \alpha$ by keeping them $<\alpha$. This is a typical case, where the level of the test approaches $\alpha$ as you increase the sample size.
- The test is only asymptotically valid, but for small sample sizes it does not keep its level (e.g. exceeds it, or perhaps ends up lower). One scenario could be that an estimate for something is used in the test statistic that is only asymptotically valid (e.g. a variance estimate), but for small sample sizes causes the test to be conservative (or anti-conservative).
- There are some other deviations from assumptions e.g. regarding the distribution etc. (e.g. non-normal residuals, using normal data methods for binomial data, assuming the normality of trial-level estimates in a meta-analysis for sparse data etc.).
-
$\begingroup$ I see, the second test I mentioned is only asymptotically valid, but I want to compare its power with a test that I constructed that is exact (works for small samples). how can I do it? $\endgroup$– noobMay 9, 2017 at 10:34
-
2$\begingroup$ The comparison is really only fair for those settings, for which the asymptotically valid test also keeps the type I error rate. For such scenarios, one can just compare them. For other scenarios, it is necessary to characterize both type I error rate and power. It is useful to understand, whether - if one, say, does not really care about exact type I error control - the asymptotic test may offer a favorable type I vs. type II error trade-off (perhaps, in some other settings an uncertain type I error rate may be unacceptable) and for what scenarios one of the tests might simply be the default. $\endgroup$– BjörnMay 9, 2017 at 11:20
-
$\begingroup$ @What do you mean by "characterize" type I and II errors? So you're saying that the comparison of the 2 tests can be reliable only asymptotically, and for small samples, we have to choose a test based on our scenario (wether we want control on type I error or we want to have more power), so we be definitive about who's test is better, right? $\endgroup$– noobMay 9, 2017 at 12:48
-
2$\begingroup$ (+1) But, though your list of deviations doesn't claim to be exhaustive, it does rather leave the impression that in the "ordinary" case the power of a test exceeds its size under the alternative. In general it depends whether or not your test's constructed to have this property, called unbiasedness. Hacking, The Logic of Statistical Inference, gives an (admittedly rather contrived) example of a generalized likelihood ratio test in which the size exceeds the power for all alternatives. A more familiar case is the chi-square test for a variance equal to a specified value, ... $\endgroup$ May 9, 2017 at 13:43
-
1$\begingroup$ ... when the rejection area is the union of two equal tail-areas - there are alternatives near the null for which the power dips a little below the size (see stats.stackexchange.com/a/224276/17230). $\endgroup$ May 9, 2017 at 13:43