Calculating necessary sample size using bootstrap

Question

I have recently come across a website (http://www.surveysystem.com/sscalc.htm) that returns the sample size given the following inputs: confidence level, confidence interval, and population. I assume this is done by rearranging a hypothesis test under a CDF, I guess using the standard normal distribution? But, if one does not believe the data reflects this particular CDF, how would you go about using bootstrapping to arrive at a version of the sample size that is data driven and not limited to the ~N(0,1) distribution?

Further, I am interested in this procedure for stratifications of a sample.

Help on either, especially the first paragraph, would be appreciated.

Answer 1

Ok, so this answer might not be exactly what you were after based on the detail of your question, but I stumbled across your question based on just the title and so this might help other people who also come across it in a similar fashion.

The only way I know of determining sample size using a bootstrap is via a power analysis approach. That is you:

1. State the null hypothesis and alternative hypothesis
2. State the alpha level (typically 5%)
3. If necessary shift the pilot study data so that you know the null hypothesis is false
4. Re-sample with replacements from the pilot study
5. Perform the test on the this sample and record the result
6. Repeat 1000 or so times to build up probability distribution
7. Count how many times the null hypothesis is rejected

With many possible "variations on a theme of..."

And that gives you the statistical power (for that sample size and that particular test), because the definition of statistical power is "probability that the test will reject the null hypothesis when the alternative hypothesis is true". So you can then vary the sample size until you achieve the desired power.

Here's an approach in R that I did based on this paper, Sample Size / Power Considerations, by Elizabeth Colantuoni.

I had two groups of non-normal, non-parametric data. A pilot study of each showed them to have differing medians and a Mann Whitney Wilcoxon test rejected the null hypothesis that they were the same, but I wanted to determine the sample size required so I could say this for "sure". Since the test already rejected the null hypothesis on the pilot data I did not see any need to shift or manipulate the data to ensure the alternative hypothesis was true.

power = function(group1.pilot, group2.pilot, reps=1000, size=10) {
    results  <- sapply(1:reps, function(r) {
        group1.resample <- sample(group1.pilot, size=size, replace=TRUE) 
        group2.resample <- sample(group2.pilot, size=size, replace=TRUE) 
        test <- wilcox.test(group1.resample, group2.resample, paired=FALSE)
        test$p.value
    })
    sum(results<0.05)/reps
}

#Find power for a sample size of 100
power(data1, data2, reps=1000, size=100)

Necessary disclaimer: I'm not a statistician and I'm still learning about bootstrapping so feedback, corrections and pointing and laughing are welcome.

Answer 2

Assuming you want to calculate the power of non normal based test like for example wilcox test, one general approach would be to simulate. The basic approach to the bootstrap for power calculation is to assume, that the effect is real count how many times, the statistical test chosen gives a statistically significant result based on the the chosen significance level over the total number of times you ran the simulation. This ratio is the power.

For the Wilcox test the below R-code shows the principle of the approach.

power = function(group1, group2, alpha=0.05, reps=1000) {
    results  <- sapply(1:reps, function(r) {
        group1.resample <- sample(group1, size=length(group1), replace=TRUE) 
        group2.resample <- sample(group2, size=length(group2), replace=TRUE) 
        test <- wilcox.test(group1.resample, group2.resample, paired=FALSE)
        test$p.value
    })
    sum(results<alpha)/reps
}

where data1 and data2 are assumed to be vectors for simplicity.
power(data1, data2, reps=1000)

Based on this approach it should be clear also how to extend the approach to more general experimental setups, such as paired data, or more groups etc. A short overview of many topics in statistics, including boostrapping, can be found in Larry Wassermans excellent book "All of statistics".

In terms of further Robust statistics Rand Wilcox book "Introduction to Robust Estimation & Hypothesis Testing" is warmly recommended, can also be quite useful in terms of looking at the source code to understand how it works (given that his WRS package contains about 1000+ functions or so).

As a side point, it would appear to me as though, for this to be useful potentially you might already have done the experiment and that leads to the issues with post hoc power analysis.