Are there any “esoteric” statistic tests with very low power?

Question

Background

In computer science, mathematics, and sometimes in other fields, “esoteric” examples cannot only be entertaining, but helpful to illustrate certain concepts, for example:

• Bogosort and Slowsort are very inefficient sorting algorithms that can be used to understand properties of algorithms, in particular when compared to other sorting algorithms.

• Esoteric programming languages demonstrate how far-reaching the concept of a programming language is and help to appreciate good programming languages.

• The Weierstraß function and Dirichlet function primarily find use to illustrate certain misconceptions about the concept of continuity.

I am currently preparing some teaching on using hypothesis tests and think that having a test with very low power (but no other flaws) would help to illustrate the concept of statistical power. (Of course, I still have to decide myself whether a given example is didactically useful for my audience or just confusing.)

Actual Question

Are there any statistical test with intentionally low power, more specifically:

• The test fits in the general framework of hypothesis tests, i.e., it works with a null hypothesis, has requirements, and returns a (correct) p value.
• It is not intended/proposed for serious application.
• It has an very low power (due to an intentional design flaw and not due to low sample or effect size).

If you can fundamentally argue that such a test cannot exist, I would also consider this a valid answer to my question. If on the other hand, a plethora of such tests exists, I am interested in the most didactically efficient one, i.e., it should be easily accessible and have a striking effect.

Note that I am not asking for a general selection of statistical mistakes (cherry picking, etc.) or similar.

What I found so far

Internet searches returned nothing for me.

Every attempt to construct something like this ended either up in some (useful) existing test or the format is not that of a regular test. For example, I thought about a test whether a population has a positive median that returns only yes if all samples are positive; but that test does not return a p value and thus does not fit within the usual test framework. If I just count the positive and negative signs as a test statistic (and compute the p values accordingly), I end up with the sign test, which is a reasonable test.

Answer 1

(Related to the comment by @Scortchi)

Suppose $X \sim N(\mu, 1)$ and we want to test the hypothesis

\begin{align*} H_0&: \mu = 0 \\ H_1&: \mu \neq 0 \end{align*}

For the sake of esetoricism, let's augment our data with an independent "coin flip" $Z \sim Bernoulli(p)$ where $p$ is known and no smaller than the significance level $\alpha$ (i.e. $p \in [\alpha, 1]$). Consider rejection regions of the form:

$$R = \left\{(X, Z) \ | \ z = 1 \ \wedge |x| > \Phi^{-1}\left(\frac{\alpha}{2p}\right) \right\}$$

By construction, this is a valid test of size $\alpha$.

\begin{align*} P(X\in R \ | \ \mu=0) &= P\left(Z=1 \ , \ |X| > \Phi^{-1}\left(\frac{\alpha}{2p}\right)\right) \\ &= P(Z=1)P\left(|X| > \Phi^{-1}\left(\frac{\alpha}{2p}\right)\right) \\ &= p\frac{\alpha}{p} = \alpha \end{align*}

The power of this test however can never be more than $p$. For instance, suppose that our observed data is $(x, z) = (1000000, 0)$. It is obvious that the null hypothesis should be rejected, but since our coin "shows tails" we fail to reject the null. Setting $p=\alpha$ leads to an even sillier example where the rejection region doesn't depend on $X$ at all, but is still a valid Rejection region with size $\alpha$.

A similar question could be given as homework by changing intersection to union in the rejection region. This region is uniformly less powerful than the one without $Z$, but is more reasonable in the sense that power doesn't have an upper bound.

Answer 2

There's a little-remarked-on corollary to the Neyman–Pearson lemma (proof in Geisser (2006), Modes of Parametric Statistical Inference, Ch 4.4): $$ \operatorname{E}\phi(X)=\alpha $$ $$ \phi(x) = \begin{cases} 0\ & \text{when $f_0(x) < kf_1(x)$} \\ 1\ & \text{when $f_0(x) > kf_1(x)$} \end{cases} $$ defines the least powerful level-$\alpha$ test, $\phi$, of the null hypothesis $H_0:$ density $f_0$ vs $H_1:$ density $f_1$ from data $x$.

From this result you can derive uniformly least powerful, locally least powerful, uniformly least powerful similar, & least powerful "totally biased" tests (I mean those with lower power under any alternative than under the null). If you already have a uniformly most powerful, &c. test, simply multiply your test statistic by -1 to maintain the partitioning of the sample space it induces while reversing the ordering of the partitions.

---

Perhaps, as @user54038 suggests, "failure of a general method of test construction" might be more interesting. Lehmann (1950), "Some principles of the theory of testing statistical hypotheses", Ann. Math. Statist., 21, 1, attributes the following example to Stein:

Let $X$ be a random variable capable of taking on the values $0, \pm 1, \pm 2$ with probabilities as indicated:

$$ \begin{array}{r c c c c c} & -2 & 2 & -1 & 1 & 0 \\ \hline \text{Hypothesis $H$:} & \frac{\alpha}{2} & \frac{\alpha}{2} & \frac{1}{2} - \alpha & \frac{1}{2} - \alpha & \alpha\\ \hline \text{Alternatives:} & pC & (1-p)C & \frac{1-C}{1-\alpha}\left(\frac{1}{2}-\alpha\right) & \frac{1-C}{1-\alpha}\left(\frac{1}{2}-\alpha\right) & \alpha\frac{1-c}{1-\alpha}\\ \end{array} $$ Here, $\alpha$, $C$, are constants $0 < \alpha \leq \frac{1}{2}$, $\frac{\alpha}{2-\alpha}< C <\alpha$, and $p$ ranges over the interval $[0,1]$.

It is desired to test the hypothesis $H$ at significance level $\alpha$. The likelihood ratio test rejects when $X=\pm2$, and hence its power is $C$ against each alternative. Since $C<\alpha$, this test is literally worse than useless, for a test with power $\alpha$ can be obtained without observing $X$ at all, simply by the use of a table of random numbers.

Note that it's the generalized likelihood ratio test he's considering, with $p$ in the role of a nuisance parameter to be maximized over. So when $X=-2$ or $X=2$, $\hat p=1$ or $\hat p=0$ respectively, & the likelihood ratio comes to $\frac{2C}{\alpha}$ in either case; for any other value of $X$ it's the lower value of $\frac{1-C}{1-\alpha}$.