6
$\begingroup$

I am trying to investigate if the proportion of successes in my population can be shown to be larger than zero. Thus, $H_0: p = 0$ and $H_A: p>0$. Since $p=0$, the prop.test in R does not work, but the bionomial test does return satisfactory results, with a p-value of $1$.

However, my main concern is the possibility of the Type II error. Given that both $p=0$ and all of my data is zero, leaving $\bar{x}=0$ and $\sigma = 0$, I am unclear of how to perform the power analysis.

I did reference this post (Power analysis for binomial data when the null hypothesis is that $p = 0$), but I don't think it has the same problem as my data presents and thus I would appreciate any direction.

$\endgroup$
1
  • 4
    $\begingroup$ You don't need or want any standard test for this. The observation of a single success will reject the null (with absolutely no uncertainty!) and observing all failures obviously should not reject the null. That's such a simple test that the power analysis will be straightforward and easy. $\endgroup$
    – whuber
    Apr 17, 2015 at 20:59

1 Answer 1

3
$\begingroup$

Before assessing the power, we have to make it clear what the test is.

This null hypothesis $H_0:p=0$ posits that successes have no chance of occurring. Observing even a single success would be convincing evidence against the null. But what if no successes (in $n$ independent trials) are observed?

For a test at level $\alpha$ you need, by definition, to have less than an $\alpha$ chance of rejecting the null when it is true. When the null is true, there are zero successes. Thus, a test may still reject the null when no successes are observed. It's just not allowed to do that more than $100\alpha$ percent of the time in the long run.

These considerations show that the test must be one of the following:

  • When one or more successes are observed, reject the null.

  • When no successes are observed, randomly reject the null with a chance $\gamma$ no greater than $\alpha$ (the "False Positive rate").

These tests are determined by the number of trials $n$ and your choice of $\gamma.$ (See the discussion at the end of this post concerning the implications of that.)

Now we can compute the power from its definition: it's the chance of rejecting the null under the alternative hypothesis. The alternative hypothesis corresponds to all nonzero values of $p$ (the success probability). In such a case, elementary probability calculations show

  • The chance of observing one or more successes in $n$ independent trials is $1 - (1-p)^n.$

  • The chance of observing zero successes and then randomly rejecting the null is $\gamma (1-p)^n.$

Thus, the chance of rejecting the null is

$$\operatorname{Power}(p;\gamma,n) = 1 - (1-p)^n + \gamma(1-p)^n = 1 - (1-\gamma)(1-p)^n.$$

For a given $n$ and $\gamma,$ these are functions of $p$ in the interval $(0,1].$ I graphed a bunch of them so you can see how they behave:

Figure

In each plot the yellow dotted lines are horizontal at a height of $\gamma=0.20$ and the yellow solid lines are the corresponding power curves. Similarly, the green colors correspond to $\gamma=0.05$ and the blue colors to $\gamma=0.$

It is follows from the formula and is visually clear in the plots that larger values of $\gamma$ lead to consistently higher power, across the board. Thus, in the usual balancing one does in selecting a test, you would want to make $\gamma$ as large as possible consistent with your need to limit the false positive rate. Evidently, then, you would choose $\gamma=\alpha.$

Thus, given your choice of test size $\alpha$ and the number of observations $n,$ the power can be made as great as $1 - (1-\alpha)(1-p)^n$ by using the test with $\gamma=\alpha.$


Note that $\gamma=0$ means your test never rejects the null when no successes are observed. All other values of $\gamma$ mean your decision is randomized: it depends not only on the observations, but also on the outcome of an independent random variable (that has nothing to do with the observations). Some people are uncomfortable using randomized tests. That's ok, but they will be forced to use the versions of this test with the lowest possible power (shown with the blue curves). That's worth pondering. (It was Jack Kiefer, I recall, who pointed out that many of the same people who refuse to use randomized tests nevertheless have no problem selecting observations randomly ;-).)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.