4
$\begingroup$

Usually all the tests we are using have increasing power as the sample size increases. But what if a test is not consistent? Is it not worthy to develop such a test? Or can it be justified to use an inconsistent test under some circumstances? In particular, as inconsistency is often due to the distributional model? An inconsistent test may still have nontrivial power, only increasing sample sizes don't pay off. Could it be reasonable to use an inconsistent test only for small sample sizes and an consistent test as his power beats the inconsistent one?

(I know that there are relevance tests. They are consistent as well but they don't have the disadvantage of consistent point hypothesis tests that show even practically irrelevant effects as significant.)

(There are inconsistent tests: E.g. for the location parameter of a shifted Cauchy distribution. As it's mean has the same distribution as a single observation, one can for all sample sizes choose the critical value from the quantile of a Cauchy distribution. But usually one would use a nonparametric consistent test.)

$\endgroup$
5
  • 1
    $\begingroup$ Do you have some specific examples in mind? $\endgroup$ Commented Jun 3, 2014 at 10:19
  • $\begingroup$ @kjetilbhalvorsen: No, it's a general question. $\endgroup$ Commented Jun 3, 2014 at 10:31
  • $\begingroup$ There was an article, I think by Agresti, that I can't find right now, that showed that tests of binomial proportions were only roughly consistent. That is, the trend in power was positive, but the power at N + 1 could be lower than at N. $\endgroup$
    – Peter Flom
    Commented Jun 3, 2014 at 10:34
  • 2
    $\begingroup$ Agresti and Coull. The local inconsistency is that the proportion is rationale but the observation is always integer. This leads to this saw teeth coverage probability of the CI. $\endgroup$ Commented Jun 3, 2014 at 10:42
  • $\begingroup$ In case the link dies: Alan Agresti and Brent A. Coull The American Statistician. Vol. 52, No. 2 (May, 1998), pp. 119-126. That result is super cool. $\endgroup$
    – Dave
    Commented Jun 25, 2020 at 2:56

1 Answer 1

5
$\begingroup$

It seems to me that inconsistency isn't of itself fatal to a test - when I'm doing some hypothesis test, my $n$ is fixed, not approaching $\infty$ - what matters is its properties at my given $n$.

It's at least possible that a test might be perfectly fine at any sample sizes I am likely to encounter (even though eventually any consistent test will do better), but the problem is that inconsistent tests often tend to have poor power at relatively modest sample sizes - it's an indication that perhaps there's information the test is not taking advantage of, that could help even at lower sample sizes.

So in practice the somewhat abstract and theoretical issue may point to a practical issue. Certainly, I wouldn't use one without performing some analysis of its behaviour at the sort of sample size or sizes I'd be using it for; it injects a solid note of caution, but I wouldn't automatically reject an inconsistent test from consideration.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.