Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'd like to do a power analysis for a single sample from binomial data, with $H_0: p = 0$, vs. $H_1: p = 0.001$, where $p$ is the proportion of successes in the population. If $0 < p <1$, I could use either the normal approximation to binomial, or $\chi^2$-test, but with $p =0$, these both fail. I'd love to know if there is a way to do this analysis. I'd very much appreciate any suggestions, comments, or references. Many thanks!

share|improve this question
So why do you not use the exact Clopper-Pearson test ? – Stéphane Laurent Oct 2 '12 at 5:22
I hope you have a really big sample! This is going to be hard to test. – Peter Flom Oct 2 '12 at 10:32
up vote 9 down vote accepted

You have a one-sided, exact alternative hypothesis $p_{1} > p_{0}$ where $p_{1} = 0.001$ and $p_{0} = 0$.

  • The first step is to identify a threshold $c$ for the number of successes such that the probability to get at least $c$ successes in a sample of size $n$ is very low under the null hypothesis (conventionally $\alpha = 0.05$). In your case, $c=1$, regardless of your particular choice for $n \geqslant 1$ and $\alpha > 0$.
  • The second step is to find out the probability to get at least $c$ successes in a sample of size $n$ under the alternative hypothesis - this is your power. Here, you need a fixed $n$ such that the Binomial distribution $\mathcal{B}(n, p_{1})$ is fully specified.

The second step in R with $n = 500$:

> n  <- 500                 # sample size
> p1 <- 0.001               # success probability under alternative hypothesis
> cc <- 1                   # threshold
> sum(dbinom(cc:n, n, p1))  # power: probability for cc or more successes given p1
[1] 0.3936211

To get an idea how the power changes with sample size, you can draw a power function: enter image description here

nn   <- 10:2000                 # sample sizes
pow  <- 1-pbinom(cc-1, nn, p1)  # corresponding power
tStr <- expression(paste("Power for ", X>0, " given ", p[1]==0.001))
plot(nn, pow, type="l", xaxs="i", xlab="sample size", ylab="power",
     lwd=2, col="blue", main=tStr, cex.lab=1.4, cex.main=1.4)

If you want to know what sample size you need to achieve at least a pre-specified power, you can use the power values calculated above. Say you want a power of at least $0.5$.

> powMin <- 0.5
> idx    <- which.min(abs(pow-powMin))  # index for value closest to 0.5
> nn[idx]     # sample size for that index
[1] 693

> pow[idx]    # power for that sample size
[1] 0.5000998

So you need a sample size of at least $693$ to achive a power of $0.5$.

share|improve this answer
According to pwr.p.test, for a power of 0.5, you need at least 677 observations. But power = 0.5 is very low! – Jessica Oct 2 '12 at 14:09
@caracal Are you using a normal approximation to get your power curve? An exact binomial power function would not be so smooth. It is actually sawtoothed which you could see if the sample size axis is magnified. I discuss this in my 2002 paper in the American Statistician coauthored with Christine Liu. Also the binomial is so highly skewed at very low p that n must be large for the normal approximation to work well. – Michael Chernick Oct 2 '12 at 15:30
@MichaelChernick No, this is from the binomial distributions, not from a normal approximation. Of course you're right that - in general - power for a binomial test is a sawtooth function that is not monotonic. But note that we have a special case here with $p_{0} = 0$. This means that the acceptance region for the alternative hypothesis always starts at 1, regardless of $n$. With a constant threshold $c = 1$, a constant $p_{1} = 0.001$, power is a strictly inreasing function of $n$. – caracal Oct 2 '12 at 19:04
@Jessica Note that pwr.p.test() uses a normal approximation, not the exact binomial distributions. Just type pwr.p.test to have a look at the source code. You'll find the calls to pnorm() indicating that an approximation is used. – caracal Oct 2 '12 at 19:26
@caracal So can I look at it this way: Under the null hypothesis the probability of success is 0 hence if you ever see a success you can reject the null hypothesis. That is why you say the threshold is 1 because if the binomial sum ever gets to 1 you can reject with a type 2 error of 0! Now under the alternative the probability of the first success on the nth trial is (1-p)$^n$$^-$$^1$ p. This probability goes to 0 as n goes to infinity. So a sequential rule going by stop when S$_n$=1 would have power 1 for any p>0. – Michael Chernick Oct 2 '12 at 19:27

You can answer this question easily with the pwr package in R.

You will need to define a significance level, power, and effect size. Typically, significance level is set to 0.05 and power is set to 0.8. Higher power will require more observations. Lower significance level will decrease power.

The effect size for proportions used in this package is Cohen's h. The cutoff for a small h is often taken to be 0.20. The actual cutoff varies by application, and might be smaller in your case. Smaller h means more observations will be required. You said your alternative is $p = 0.001$. That is very small

> ES.h(.001, 0)
[1] 0.0632561

But we can still proceed.

 > pwr.p.test(sig.level=0.05, power=.8, h = ES.h(.001, 0), alt="greater", n = NULL)

 proportion power calculation for binomial distribution (arcsine transformation) 

          h = 0.0632561
          n = 1545.124
  sig.level = 0.05
      power = 0.8
alternative = greater

Using these values, you need at least 1546 observations.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.