# Calculating necessary sample size using bootstrap

I have recently come across a website (http://www.surveysystem.com/sscalc.htm) that returns the sample size given the following inputs: confidence level, confidence interval, and population. I assume this is done by rearranging a hypothesis test under a CDF, I guess using the standard normal distribution? But, if one does not believe the data reflects this particular CDF, how would you go about using bootstrapping to arrive at a version of the sample size that is data driven and not limited to the ~N(0,1) distribution?

Further, I am interested in this procedure for stratifications of a sample.

Help on either, especially the first paragraph, would be appreciated.

-
It appears that this company is providing a "calculator" to produce a required estimate of some unspecified parameter of a population distribution (unspecified) of a given fixed width. I would not trust it. There are so many reputable programs developed by statisticians that can do sample size determination for a variety of problems and do it correctly (e.g. nQuery Advisor, SiZ, proc power in SAS, power and precision etc.). This company can't even get their definitions straight. –  Michael Chernick Sep 22 '12 at 12:55
They confuse confidence interval with margin of error and provide several definitons that are either wrong or incorrectly explained. To do a sample size calculation for sample fixed width confidence intervals you need to know the parametric population model, the parameter to be estimated, the variance of an individual random observayion and the confidence level. It is so bad that it is really hard to tell what they are doing and whether they are doing it correctly. They have a box for population but don't let you specify it. –  Michael Chernick Sep 22 '12 at 13:03
Presumably you are correct in assuming that they are doing this for a normal population but they don't even let you specify the variance. so maybe they assume it is 1 as you suggest in your question. What parameter are they trying to estimate with the confidence interval? I would presume it is the population mean. But even that is not mentioned. So my advice is that you can probably find what you are looking for in the software packages I mentioned as long as you are willing to use a parametric model (perhaps only a select few parametric models will be available. –  Michael Chernick Sep 22 '12 at 13:08
Nevertheless Kirk you may have an interesting question here if it is posed a little differently. Is it that you want to do sample size determination in a nonparametric framework for confidence intervals? If so you can do this for estimating certain parameters such as a population median using rank statistics without needing to use bootstrap. If you have a specific problem where you think the bootstrap would be needed and you want to avoid parametric assumptions please provide one and I can try to give you an answer for that situation. –  Michael Chernick Sep 22 '12 at 13:15
I agree with Michael Chernick. This calculator is only as good as marketing and low quality polls go, for the lack of a better curse word :). What they sweep under the carpet and totally failed to explain is that their tool is only intended to work with proportions -- that's the percentage in the lower box, and the upper box probably assumes 50% that requires the largest sample size. If you know what the bootstrap is, you are already overqualified in terms of using these crutches, assuming your know one or more of R, Stata or SAS, and can do the power analysis yourself using these packages. –  StasK Sep 22 '12 at 18:27
show 1 more comment

Ok, so this answer might not be exactly what you were after based on the detail of your question, but I stumbled across your question based on just the title and so this might help other people who also come across it in a similar fashion.

The only way I know of determining sample size using a bootstrap is via a power analysis approach. That is you:

1. State the null hypothesis and alternative hypothesis
2. State the alpha level (typically 5%)
3. If necessary shift the pilot study data so that you know the null hypothesis is false
4. Re-sample with replacements from the pilot study
5. Perform the test on the this sample and record the result
6. Repeat 1000 or so times to build up probability distribution
7. Count how many times the null hypothesis is rejected

With many possible "variations on a theme of..."

And that gives you the statistical power (for that sample size and that particular test), because the definition of statistical power is "probability that the test will reject the null hypothesis when the alternative hypothesis is true". So you can then vary the sample size until you achieve the desired power.

Here's an approach in R that I did based on this paper, Sample Size / Power Considerations, by Elizabeth Colantuoni.

I had two groups of non-normal, non-parametric data. A pilot study of each showed them to have differing medians and a Mann Whitney Wilcoxon test rejected the null hypothesis that they were the same, but I wanted to determine the sample size required so I could say this for "sure". Since the test already rejected the null hypothesis on the pilot data I did not see any need to shift or manipulate the data to ensure the alternative hypothesis was true.

power = function(group1.pilot, group2.pilot, reps=1000, size=10) {
results  <- sapply(1:reps, function(r) {
group1.resample <- sample(group1.pilot, size=size, replace=TRUE)
group2.resample <- sample(group2.pilot, size=size, replace=TRUE)
test <- wilcox.test(group1.resample, group2.resample, paired=FALSE)
test\$p.value
})
sum(results<0.05)/reps
}

#Find power for a sample size of 100
power(data1, data2, reps=1000, size=100)


Necessary disclaimer: I'm not a statistician and I'm still learning about bootstrapping so feedback, corrections and pointing and laughing are welcome.

-