# Are inconsistent tests bearable?

Usually all the tests we are using have increasing power as the sample size increases. But what if a test is not consistent? Is it not worthy to develop such a test? Or can it be justified to use an inconsistent test under some circumstances? In particular, as inconsistency is often due to the distributional model? An inconsistent test may still have nontrivial power, only increasing sample sizes don't pay off. Could it be reasonable to use an inconsistent test only for small sample sizes and an consistent test as his power beats the inconsistent one?

(I know that there are relevance tests. They are consistent as well but they don't have the disadvantage of consistent point hypothesis tests that show even practically irrelevant effects as significant.)

(There are inconsistent tests: E.g. for the location parameter of a shifted Cauchy distribution. As it's mean has the same distribution as a single observation, one can for all sample sizes choose the critical value from the quantile of a Cauchy distribution. But usually one would use a nonparametric consistent test.)

• Do you have some specific examples in mind? Commented Jun 3, 2014 at 10:19
• @kjetilbhalvorsen: No, it's a general question. Commented Jun 3, 2014 at 10:31
• There was an article, I think by Agresti, that I can't find right now, that showed that tests of binomial proportions were only roughly consistent. That is, the trend in power was positive, but the power at N + 1 could be lower than at N. Commented Jun 3, 2014 at 10:34
• Agresti and Coull. The local inconsistency is that the proportion is rationale but the observation is always integer. This leads to this saw teeth coverage probability of the CI. Commented Jun 3, 2014 at 10:42
• In case the link dies: Alan Agresti and Brent A. Coull The American Statistician. Vol. 52, No. 2 (May, 1998), pp. 119-126. That result is super cool.
– Dave
Commented Jun 25, 2020 at 2:56

It seems to me that inconsistency isn't of itself fatal to a test - when I'm doing some hypothesis test, my $$n$$ is fixed, not approaching $$\infty$$ - what matters is its properties at my given $$n$$.