In computer science, mathematics, and sometimes in other fields, “esoteric” examples cannot only be entertaining, but helpful to illustrate certain concepts, for example:

  • Bogosort and Slowsort are very inefficient sorting algorithms that can be used to understand properties of algorithms, in particular when compared to other sorting algorithms.

  • Esoteric programming languages demonstrate how far-reaching the concept of a programming language is and help to appreciate good programming languages.

  • The Weierstraß function and Dirichlet function primarily find use to illustrate certain misconceptions about the concept of continuity.

I am currently preparing some teaching on using hypothesis tests and think that having a test with very low power (but no other flaws) would help to illustrate the concept of statistical power. (Of course, I still have to decide myself whether a given example is didactically useful for my audience or just confusing.)

Actual Question

Are there any statistical test with intentionally low power, more specifically:

  • The test fits in the general framework of hypothesis tests, i.e., it works with a null hypothesis, has requirements, and returns a (correct) p value.
  • It is not intended/proposed for serious application.
  • It has an very low power (due to an intentional design flaw and not due to low sample or effect size).

If you can fundamentally argue that such a test cannot exist, I would also consider this a valid answer to my question. If on the other hand, a plethora of such tests exists, I am interested in the most didactically efficient one, i.e., it should be easily accessible and have a striking effect.

Note that I am not asking for a general selection of statistical mistakes (cherry picking, etc.) or similar.

What I found so far

Internet searches returned nothing for me.

Every attempt to construct something like this ended either up in some (useful) existing test or the format is not that of a regular test. For example, I thought about a test whether a population has a positive median that returns only yes if all samples are positive; but that test does not return a p value and thus does not fit within the usual test framework. If I just count the positive and negative signs as a test statistic (and compute the p values accordingly), I end up with the sign test, which is a reasonable test.

  • 2
    $\begingroup$ Being more mathematical, "esoteric" examples (which abound) tend to be specific counterexamples to popular misunderstandings; a number of textbooks contain such examples. As it stands, your question is essentially a "big list" type question and so is too broad (though you should note that several users have concluded the question is unclear); if you can clarify your question and narrow its scope it may fit the site better. $\endgroup$
    – Glen_b
    Feb 17, 2019 at 4:00
  • 1
    $\begingroup$ Low power compared to what? Lehmann gave an example of a generalized likelihood-ratio test that had lower power under any alternative hypothesis than under the null. $\endgroup$ Feb 17, 2019 at 17:03
  • 2
    $\begingroup$ Any of the silly estimators to which you apply Rao-Blackwellization could be used as a test statistic. For example, there's the first observation in the sample, used as an estimator of the mean. When Rao-Blackwellized, you obtain the sample mean. I had to do many exercises like this in class. Anyway, this statistic could be used instead of the sample mean in something like a $t$ test. But no, I can't think of anything directly in the form you're looking for, or I'd be writing an answer, not a comment. But there must be something, illustrating failure of a general method for test construction. $\endgroup$
    – user54038
    Feb 17, 2019 at 18:31
  • 1
    $\begingroup$ I'll dig out the Lehmann paper when I'm at a computer. The power of a test under the null is just the size of the test. $\endgroup$ Feb 17, 2019 at 21:09
  • 4
    $\begingroup$ An example test used in a class I was a student in (many years ago) was "roll a fair 20 sided die and reject if you roll a 1" (as part of a discussion of power curves). This of course completely ignores the data, but is a "valid" test in that it doesn't have higher than the desired type I error rate (which was 5% in the context the example was given in) . $\endgroup$
    – Glen_b
    Feb 17, 2019 at 23:15

2 Answers 2


(Related to the comment by @Scortchi)

Suppose $X \sim N(\mu, 1)$ and we want to test the hypothesis

\begin{align*} H_0&: \mu = 0 \\ H_1&: \mu \neq 0 \end{align*}

For the sake of esetoricism, let's augment our data with an independent "coin flip" $Z \sim Bernoulli(p)$ where $p$ is known and no smaller than the significance level $\alpha$ (i.e. $p \in [\alpha, 1]$). Consider rejection regions of the form:

$$R = \left\{(X, Z) \ | \ z = 1 \ \wedge |x| > \Phi^{-1}\left(\frac{\alpha}{2p}\right) \right\}$$

By construction, this is a valid test of size $\alpha$.

\begin{align*} P(X\in R \ | \ \mu=0) &= P\left(Z=1 \ , \ |X| > \Phi^{-1}\left(\frac{\alpha}{2p}\right)\right) \\ &= P(Z=1)P\left(|X| > \Phi^{-1}\left(\frac{\alpha}{2p}\right)\right) \\ &= p\frac{\alpha}{p} = \alpha \end{align*}

The power of this test however can never be more than $p$. For instance, suppose that our observed data is $(x, z) = (1000000, 0)$. It is obvious that the null hypothesis should be rejected, but since our coin "shows tails" we fail to reject the null. Setting $p=\alpha$ leads to an even sillier example where the rejection region doesn't depend on $X$ at all, but is still a valid Rejection region with size $\alpha$.

A similar question could be given as homework by changing intersection to union in the rejection region. This region is uniformly less powerful than the one without $Z$, but is more reasonable in the sense that power doesn't have an upper bound.

  • 2
    $\begingroup$ (+1) Closely related because having a one-dimensional ancillary statistic $S$, you can dispense with the coin flip by letting $Z=\boldsymbol{1}(S<F_S^{-1}(p))$, where $F_S(\cdot)$ is the distribution function of $S$. $\endgroup$ Feb 17, 2019 at 23:39

There's a little-remarked-on corollary to the Neyman–Pearson lemma (proof in Geisser (2006), Modes of Parametric Statistical Inference, Ch 4.4): $$ \operatorname{E}\phi(X)=\alpha $$ $$ \phi(x) = \begin{cases} 0\ & \text{when $f_0(x) < kf_1(x)$} \\ 1\ & \text{when $f_0(x) > kf_1(x)$} \end{cases} $$ defines the least powerful level-$\alpha$ test, $\phi$, of the null hypothesis $H_0:$ density $f_0$ vs $H_1:$ density $f_1$ from data $x$.

From this result you can derive uniformly least powerful, locally least powerful, uniformly least powerful similar, & least powerful "totally biased" tests (I mean those with lower power under any alternative than under the null). If you already have a uniformly most powerful, &c. test, simply multiply your test statistic by -1 to maintain the partitioning of the sample space it induces while reversing the ordering of the partitions.

Perhaps, as @user54038 suggests, "failure of a general method of test construction" might be more interesting. Lehmann (1950), "Some principles of the theory of testing statistical hypotheses", Ann. Math. Statist., 21, 1, attributes the following example to Stein:

Let $X$ be a random variable capable of taking on the values $0, \pm 1, \pm 2$ with probabilities as indicated:

$$ \begin{array}{r c c c c c} & -2 & 2 & -1 & 1 & 0 \\ \hline \text{Hypothesis $H$:} & \frac{\alpha}{2} & \frac{\alpha}{2} & \frac{1}{2} - \alpha & \frac{1}{2} - \alpha & \alpha\\ \hline \text{Alternatives:} & pC & (1-p)C & \frac{1-C}{1-\alpha}\left(\frac{1}{2}-\alpha\right) & \frac{1-C}{1-\alpha}\left(\frac{1}{2}-\alpha\right) & \alpha\frac{1-c}{1-\alpha}\\ \end{array} $$ Here, $\alpha$, $C$, are constants $0 < \alpha \leq \frac{1}{2}$, $\frac{\alpha}{2-\alpha}< C <\alpha$, and $p$ ranges over the interval $[0,1]$.

It is desired to test the hypothesis $H$ at significance level $\alpha$. The likelihood ratio test rejects when $X=\pm2$, and hence its power is $C$ against each alternative. Since $C<\alpha$, this test is literally worse than useless, for a test with power $\alpha$ can be obtained without observing $X$ at all, simply by the use of a table of random numbers.

Note that it's the generalized likelihood ratio test he's considering, with $p$ in the role of a nuisance parameter to be maximized over. So when $X=-2$ or $X=2$, $\hat p=1$ or $\hat p=0$ respectively, & the likelihood ratio comes to $\frac{2C}{\alpha}$ in either case; for any other value of $X$ it's the lower value of $\frac{1-C}{1-\alpha}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.