6
$\begingroup$

Background:

A group at work is sampling 1,000 customers to contact and from that point, determining if the effort is worthwhile or not. I wanted to see if this (pretty much) arbitrary sample size value was at all "good enough".

If we assume that the true proportion of successes in a population is 0.016 (1.6%) how large a sample size would I need to take in order to get a confidence interval margin of error "half width") of say 0.005 (0.5%)? Here is my approach in R:

install.packages("Hmisc")
library(Hmisc)


target.halfWidth<-0.005
sims<-25000 #number of draws from binomial to perform


p<-0.016 #true proportion
n<-seq(from=500, to=5000, by=100) #number of samples

#hold results
results<-matrix(numeric(0), length(n),2)

#loop through desired sample size options
for (i in 1: length(n))
{

x<-rbinom(sims,n[i],p) #draws from binomial with p and n 
ci<-binconf(x,n[i],method="asymptotic",alpha=0.1) #normal theory 90% CI
half_width<-ci[,3]-ci[,1] #half width of CI

#need the number where the half width is within the target range
prob.halfWidth<-length(half_width[half_width<target.halfWidth])/sims

#store results
results[i,1]<-n[i]
results[i,2]<-prob.halfWidth
}

#plot
plot(results[,2],results[,1],type="b")
results 

This simulation showed that we would need 2,200 samples to be 95% confident that a 90% CI would be at most 0.005.

Questions:

1) Is this a proper method?

2) Are there better ways?

3) What advice can you give is there are finite samples of some sub-populations? Say that we want to know how many samples to take of populations where there are not "a lot" of customers to choose from. Maybe there is only 5,000 of a certain group, should we not be able to take less of them to make a determination compared to a group where there are 50,000 to choose from?

Thanks!!

Add after MansT answer:

1) Would it make sense, under my scenario with the simulation draws to add a step where this line:

prob.halfWidth<-length(half_width[half_width<target.halfWidth])/sims

only increments the numerator when the resulting CI also contains the true p (i.e. 0.016)?

2) Under your code, would it also be appropriate when dealing with a finite sample to add the finite population corrector FPC to your line:

halfWidth<-qnorm(0.95)*sqrt(p.est*(1-p.est)/n)

3) I am not sure the formula for a CI from the hypergeometric, but perhaps I could replace my code line

ci<-binconf(x,n[i],method="asymptotic",alpha=0.1) #normal theory 90% CI 

with Sprop function in R?

$\endgroup$
5
$\begingroup$

(1) Yes.

(2) Yes. There are only $n+1$ possible outcomes for a binomial random variable, so it is possible to look at what happens for each possible outcome - in fact this is faster than simulating lots and lots of outcomes!

Let $X$ be the number of "successes" among the $n$ customers and let $\hat{p}=X/n$. The confidence interval is $\hat{p}\pm z_{\alpha/2}\sqrt{\hat{p}(1-\hat{p})/n}$, so the halfwidth is $z_{\alpha/2}\sqrt{\hat{p}(1-\hat{p})/n}$. Thus we want to compute $P(z_{\alpha/2}\sqrt{\hat{p}(1-\hat{p})/n}\leq 0.005)$. In R, we can do this as follows:

target.halfWidth<-0.005

p<-0.016 #true proportion
n.vec<-seq(from=1000, to=3000, by=100) #number of samples

# Vector to store results
prob.hw<-rep(NA,length(n.vec))

# Loop through desired sample size options
for (i in 1: length(n.vec))
{
n<-n.vec[i]

# Look at all possible outcomes
x<-0:n
p.est<-x/n

# Compute halfwidth for each option
halfWidth<-qnorm(0.95)*sqrt(p.est*(1-p.est)/n)

# What is the probability that the halfwidth is less than 0.005?
prob.hw[i]<-sum({halfWidth<=target.halfWidth}*dbinom(x,n,p))
}

# Plot results
plot(n.vec,prob.hw,type="b")
abline(0.95,0,col=2)

# Get the minimal n required
n.vec[min(which(prob.hw>=0.95))]

The answer is $n=2200$ in this case as well.

Finally, it is usually a good idea to verify that the asymptotic normal approximation interval actually gives the desired coverage. In R, we can compute the coverage probability (i.e. the actual confidence level) as:

p<-0.016
n<-2200
x<-0:n
p.est<-x/n
halfWidth<-qnorm(0.95)*sqrt(p.est*(1-p.est)/n)
# Coverage probability
sum({abs(p-p.est)<=halfWidth}*dbinom(x,n,p))

Different $p$ give different coverages. For $p$ around $0.015$, the actual confidence level of the nominal $90\%$ interval seems to be about $89\%$ in general, which I presume is fine for your purposes.

(3) When you sample from a finite population, the number of successes is not binomial but hypergeometric. If the population is large compared to your sample size, the binomial works just fine as an approximation. If you sample 1000 out of 5000, say, it does not. Have a look at confidence intervals for proportions based on the hypergeometric distribution!

Answers to additional questions:

Let $(p_L,p_U)$ be the confidence interval.

1) In that case you are no longer computing $P(p_L-p_U\leq0.01)$ but $$P\Big(p_L-p_U\leq0.01~\mbox{and}~p\in(p_L,p_U)\Big),$$ i.e. the probability that the length of intervals that actually contain $p$ is at most 0.01. This may be an interesting quantity, depending on what you're interested in...

2) Maybe, but probably not. If the population size is large compared to the sample size you don't need it, and if it's not then the binomial distribution is not appropriate to begin with!

3) Sprop seems to contain confidence intervals based on the hypergeometric intervals, so that should work just fine.

$\endgroup$
0
$\begingroup$

You could use the normal approximation to the binomial to determine the width of the confidence interval. The variance is $p(1-p)/n$. The standard deviation is $\sqrt{p(1-p)/n}$. Since you expect $p=0.016$ apply $p=0.02$ in the formula.

The interval with width 4 sigma from mean-2 sigma to mean + two sigma is approximately a 95% two-sided confidence interval.

So let $4\sigma =0.01$ (half width 0.005). So $4 \sqrt{0.02(0.98)/n}=0.01$ or $16 \cdot0.0196/n=0.01$ or $n=16\cdot 0.01960/0.0001= 16\cdot 196=3136.$ So you need about 3100 to have a 95% confidence interval with width 0.01.

$\endgroup$
  • $\begingroup$ I have wondered about simple formula like this - what power level is implied? Is that just ignored? $\endgroup$ – B_Miner Jun 12 '12 at 13:41
  • 1
    $\begingroup$ This is a confidence interval rather than a hypothesis test. For hypothesis testing you would use power instead of interval width/half-width. What is done is that you pick the difference worth detecting and choose the sample size that makes the probability of exceeding the critical value the specified power ( say 0.95) when that alternative distribution is the true one. $\endgroup$ – Michael R. Chernick Jun 12 '12 at 14:32
  • $\begingroup$ It should probably be noted that this gives an approximation of when the expected length of the interval is 0.01 (for the 95% interval and not 90%). This does not (directly) tell us what the probability of the length being at most 0.01 is. $\endgroup$ – MånsT Jun 12 '12 at 17:56
  • $\begingroup$ @MansT I think you are wrong. The interval length is not random. It is fixed at 0.01 by taking n sufficeintly large. What is random is whether or not the random interval contains the true p. That is assuming we know in advance that p<.02. I would not use the sample estimate of p in deciding on n. To do this cleanly we can bound p(1-p) above by 1/4 and use that to select an n that is guaranteed to give an interval with width less than 0.01 and having at least 95% probability of including the true p. $\endgroup$ – Michael R. Chernick Jun 12 '12 at 18:11
  • $\begingroup$ @MichaelChernick: Typically $\hat{p}$ (which is random) is used instead of $p$ in the confidence interval. Thus the interval width, $2z_{\alpha/2}\sqrt{\hat{p}\hat{q}/n}$ is also random (although the variance decreases as $n\rightarrow\infty$). Or did I misunderstand you somehow? $\endgroup$ – MånsT Jun 12 '12 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.