2
$\begingroup$

I am having a bit of confusion with hypothesis testing basics, could you please help me clear it?

Suppose we are testing a simple hypothesis $H_0:\theta=\theta_0$ against another simple hypothesis $H_1:\theta=\theta_1$ via an iid sample $X_1,...,X_n$. Let also the family of underlying distribuions be absolutely continous with density $p_\theta(x)$ (so that we don't have to worry about randomization). Then the Neyman - Pearson criterion at a significance level $\alpha$, which is the most powerful criterion with a given significance level, states that the hypothesis $H_0$ should be rejected if $$ L(X)=\prod_{i=1}^n\frac{p_{\theta_1}(x_i)}{p_{\theta_0}(x_i)}>c $$ for some $c$. It also follows from the proof of Neyman - Pearson lemma that $c$ should be chosen so that the I-type error probability is exactly $\alpha$: $$ P(L(X)>c|H_0)=\alpha. $$ On the other hand, I have recently encountered a notion of criterion conststency. Since most of the available literature is of an applied statistical flavour, I can't find a proper definition of consistency, but I am assuming that it is analogous to the definition of estimator consistency: $$ \lim_{n\to\infty}P(L(X)>c|H_0)=0. $$ Thus, it follows from the definition that the Neyman - Pearson criterion is not consistent, yet most powerful and just seems like the best thing you can do.

TL/DR: is my definition of consistency correct and, if so, is the Neyman-Pearson criterion inconsistent? If yes, why is it a good thing to use? Isn't consistency more important that power?

Improve this question
$\endgroup$
5
  • $\begingroup$ Type-I error probability is $P(L(X)>{c}|H_0)=\alpha$. A notion of consistency that I am familiar with in the context of testing is test consistency, which says that $\lim_{n\to\infty}P(L(X)>{c}|H_1)=1$. Do maybe have a reference for "criterion consistency"? $\endgroup$
    – Christoph Hanck
    Commented Mar 17, 2016 at 13:54
  • $\begingroup$ @ChristophHanck I'm sorry, I was thinking of the correct thing but typed some nonsense. As for your definition of test consistency - thanks a lot for this, this is basically what I wanted. Could you provide some reference to the definition? $\endgroup$
    – Vossler
    Commented Mar 17, 2016 at 14:01
  • $\begingroup$ See e.g. en.wikipedia.org/wiki/Consistency_(statistics) But your definition of criterion consistency says something else, namely that the probability of a type-I error goes to 0 asymptotically. Unless $c$ (suitably) depends on (i.e., increases with) $n$, this is not going to be possible, as the test statistic is a nondegenerate r.v. under $H_0$ even as $n\to\infty$. $\endgroup$
    – Christoph Hanck
    Commented Mar 17, 2016 at 14:04
  • $\begingroup$ @ChristophHanck Thanks a lot! Your definition is most probably the right one since I didn't see a proper definition anywhere and tried to guess what this could mean (by carrying it over from estimator consistency). I see now that my guess was wrong, so case pretty much closed. $\endgroup$
    – Vossler
    Commented Mar 17, 2016 at 14:08
  • $\begingroup$ Good, but there is indeed a literature on tests that apply ever more challenging critical values $c_n$ as $n\to\infty$ so as to bring down the type-I error probability asymptotically, but I struggle to find a good reference at the moment. Think, e.g., consistent model selection criteria such as BIC which are such that $\lim P(\text{model A is selected}|\text{model A is the true model})=1$. $\endgroup$
    – Christoph Hanck
    Commented Mar 17, 2016 at 14:31

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.