So basically I have the following scenario:

I have developed a new way for conditional independence testing, and am bench-marking is against other methods.

To do this, I test the power in scenarios, where the null-hypothesis of conditional independence should be rejected (because it is false), however the tests are very complex (low signal to noise ratio), and hence for some sample sizes, the power is (very) low. Which made me wonder:

If, say, I fix the confidence (desired type 1 error) at 5%, would it ever make sense to have a power that is below 5%?

Because if the test gives you at least 5% type 1 errors, shouldn't it reject at least 5% of the cases, where the null is actually false (as a lower bound).

Logically, it makes complete sense to me - but I couldn't find any mathematical reasoning supporting this. Does anyone know how to derive this lower bound? (or why its not actually a lower bound)

  • It is not a lower bound because a test may be "biased". See, e.g., this thread for some further discussion: stats.stackexchange.com/questions/279162/… – Christoph Hanck Aug 31 '17 at 6:28
  • There will always be a probability that a practical case occurs in which a benchmark test gives you less rejections for $H_1$ than $H_0$. Especially, if you have a biased test (as Christoph points out), and actually do not determine the number of rejections for $H_0$ but simply assume that it is $\alpha$ then you may consistently get less rejections for $H_1$ than a (theoretic) $\alpha$-level. If you get extreme deviations and doubt yourself, then you might look at a plot of the power function and see if it is actually minimum for the parameters associated with the $H_0$. – Martijn Weterings Aug 31 '17 at 13:31
  • I'm not sure if that 100% addresses what I mean and I don't get it. Basically, my question is, can p(reject $H_0$ | $H_0$ false) ever be lower than p(reject $H_0$ | $H_0$ true). What I mean is of course in the expectation (clearly this could occur as a result of randomness). Is that what you two are adressing as well? – Sam Aug 31 '17 at 14:30
  • The randomness is 1 of the points that I addressed. One other case is that you may get the inequality if you base the expectations on two different measures, with p(reject H_0 | H_0 false) a theoretic value (which may be false if assumptions are not met) and p(reject H_0 | H_0 true) an experimentally obtained value (which may also be false if randomization is not correctly done, or indeed a result of randomness). And another case is if indeed p(reject H_0 | H_a true) < p(reject H_0 | H_0 true) which means the test is badly created (this is hypothetical and why I asked the origin of the Q). – Martijn Weterings Aug 31 '17 at 14:41
  • Actually, for specific alternative hypotheses, the rejection rate might be $<\alpha$ even if the minimum of rejection is inside the range of $H_0$. The rejection rate is a function of the underlying parameters. If this rejection rate is not a continuously increasing function for larger distance from $H_0$ and, for instance has multiple minima, then it might be that you obtain a lower rejection rate. Hypothetical example. If you observe x^2 and wish to establish that $H_0: 0.9<x<1.1$ with minimum type I error rate $\alpha$, then possibly you have a lower rejection than $\alpha$ if $x=-1$. – Martijn Weterings Aug 31 '17 at 14:49

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.