Does the minimum of the power of a test always have to equal $\alpha$ the significance level? I'm comparing the power of 2 tests, one always has its minimum at $\alpha$, while the other have its minimum somtimes below $\alpha$, other times it's above $\alpha$, it depends on the sample size. Is this behavior normal? what does this say about the second test?
Tests will under the null hypothesis or with an effect that is very small (i.e. not very far away from the null hypothesis) typically have a power close to their significance level. Once you move into the interior of parameter set that constitutes the null hypothesis (if there is such a thing), their "power" will normally be lower than that.
Reasons for deviations from this include
- The test is conservative and does not fully exhaust the significance level. This for example is known to be the case for Fisher's exact test and other exact methods for small sample sizes - with a small discrete outcome space, exact methods exact methods keep the significance level $\leq \alpha$ by keeping them $<\alpha$. This is a typical case, where the level of the test approaches $\alpha$ as you increase the sample size.
- The test is only asymptotically valid, but for small sample sizes it does not keep its level (e.g. exceeds it, or perhaps ends up lower). One scenario could be that an estimate for something is used in the test statistic that is only asymptotically valid (e.g. a variance estimate), but for small sample sizes causes the test to be conservative (or anti-conservative).
- There are some other deviations from assumptions e.g. regarding the distribution etc. (e.g. non-normal residuals, using normal data methods for binomial data, assuming the normality of trial-level estimates in a meta-analysis for sparse data etc.).