I have recently come across a website (http://www.surveysystem.com/sscalc.htm) that returns the sample size given the following inputs: confidence level, confidence interval, and population. I assume this is done by rearranging a hypothesis test under a CDF, I guess using the standard normal distribution? But, if one does not believe the data reflects this particular CDF, how would you go about using bootstrapping to arrive at a version of the sample size that is data driven and not limited to the ~N(0,1) distribution?

Further, I am interested in this procedure for stratifications of a sample.

Help on either, especially the first paragraph, would be appreciated.

  • 2
    $\begingroup$ It appears that this company is providing a "calculator" to produce a required estimate of some unspecified parameter of a population distribution (unspecified) of a given fixed width. I would not trust it. There are so many reputable programs developed by statisticians that can do sample size determination for a variety of problems and do it correctly (e.g. nQuery Advisor, SiZ, proc power in SAS, power and precision etc.). This company can't even get their definitions straight. $\endgroup$ – Michael R. Chernick Sep 22 '12 at 12:55
  • 2
    $\begingroup$ They confuse confidence interval with margin of error and provide several definitons that are either wrong or incorrectly explained. To do a sample size calculation for sample fixed width confidence intervals you need to know the parametric population model, the parameter to be estimated, the variance of an individual random observayion and the confidence level. It is so bad that it is really hard to tell what they are doing and whether they are doing it correctly. They have a box for population but don't let you specify it. $\endgroup$ – Michael R. Chernick Sep 22 '12 at 13:03
  • $\begingroup$ Presumably you are correct in assuming that they are doing this for a normal population but they don't even let you specify the variance. so maybe they assume it is 1 as you suggest in your question. What parameter are they trying to estimate with the confidence interval? I would presume it is the population mean. But even that is not mentioned. So my advice is that you can probably find what you are looking for in the software packages I mentioned as long as you are willing to use a parametric model (perhaps only a select few parametric models will be available. $\endgroup$ – Michael R. Chernick Sep 22 '12 at 13:08
  • $\begingroup$ Nevertheless Kirk you may have an interesting question here if it is posed a little differently. Is it that you want to do sample size determination in a nonparametric framework for confidence intervals? If so you can do this for estimating certain parameters such as a population median using rank statistics without needing to use bootstrap. If you have a specific problem where you think the bootstrap would be needed and you want to avoid parametric assumptions please provide one and I can try to give you an answer for that situation. $\endgroup$ – Michael R. Chernick Sep 22 '12 at 13:15
  • 2
    $\begingroup$ I agree with Michael Chernick. This calculator is only as good as marketing and low quality polls go, for the lack of a better curse word :). What they sweep under the carpet and totally failed to explain is that their tool is only intended to work with proportions -- that's the percentage in the lower box, and the upper box probably assumes 50% that requires the largest sample size. If you know what the bootstrap is, you are already overqualified in terms of using these crutches, assuming your know one or more of R, Stata or SAS, and can do the power analysis yourself using these packages. $\endgroup$ – StasK Sep 22 '12 at 18:27

Ok, so this answer might not be exactly what you were after based on the detail of your question, but I stumbled across your question based on just the title and so this might help other people who also come across it in a similar fashion.

The only way I know of determining sample size using a bootstrap is via a power analysis approach. That is you:

  1. State the null hypothesis and alternative hypothesis
  2. State the alpha level (typically 5%)
  3. If necessary shift the pilot study data so that you know the null hypothesis is false
  4. Re-sample with replacements from the pilot study
  5. Perform the test on the this sample and record the result
  6. Repeat 1000 or so times to build up probability distribution
  7. Count how many times the null hypothesis is rejected

With many possible "variations on a theme of..."

And that gives you the statistical power (for that sample size and that particular test), because the definition of statistical power is "probability that the test will reject the null hypothesis when the alternative hypothesis is true". So you can then vary the sample size until you achieve the desired power.

Here's an approach in R that I did based on this paper, Sample Size / Power Considerations, by Elizabeth Colantuoni.

I had two groups of non-normal, non-parametric data. A pilot study of each showed them to have differing medians and a Mann Whitney Wilcoxon test rejected the null hypothesis that they were the same, but I wanted to determine the sample size required so I could say this for "sure". Since the test already rejected the null hypothesis on the pilot data I did not see any need to shift or manipulate the data to ensure the alternative hypothesis was true.

power = function(group1.pilot, group2.pilot, reps=1000, size=10) {
    results  <- sapply(1:reps, function(r) {
        group1.resample <- sample(group1.pilot, size=size, replace=TRUE) 
        group2.resample <- sample(group2.pilot, size=size, replace=TRUE) 
        test <- wilcox.test(group1.resample, group2.resample, paired=FALSE)

#Find power for a sample size of 100
power(data1, data2, reps=1000, size=100)

Necessary disclaimer: I'm not a statistician and I'm still learning about bootstrapping so feedback, corrections and pointing and laughing are welcome.


Assuming you want to calculate the power of non normal based test like for example wilcox test, one general approach would be to simulate. The basic approach to the bootstrap for power calculation is to assume, that the effect is real count how many times, the statistical test chosen gives a statistically significant result based on the the chosen significance level over the total number of times you ran the simulation. This ratio is the power.

For the Wilcox test the below R-code shows the principle of the approach.

power = function(group1, group2, alpha=0.05, reps=1000) {
    results  <- sapply(1:reps, function(r) {
        group1.resample <- sample(group1, size=length(group1), replace=TRUE) 
        group2.resample <- sample(group2, size=length(group2), replace=TRUE) 
        test <- wilcox.test(group1.resample, group2.resample, paired=FALSE)

where data1 and data2 are assumed to be vectors for simplicity.
power(data1, data2, reps=1000)

Based on this approach it should be clear also how to extend the approach to more general experimental setups, such as paired data, or more groups etc. A short overview of many topics in statistics, including boostrapping, can be found in Larry Wassermans excellent book "All of statistics".

In terms of further Robust statistics Rand Wilcox book "Introduction to Robust Estimation & Hypothesis Testing" is warmly recommended, can also be quite useful in terms of looking at the source code to understand how it works (given that his WRS package contains about 1000+ functions or so).

As a side point, it would appear to me as though, for this to be useful potentially you might already have done the experiment and that leads to the issues with post hoc power analysis.

  • 1
    $\begingroup$ Please refrain from posting comments as answers. If you lack the reputation to post comments, contribute to the website in other forms (asking questions, providing valid answers) until you accrue enough points. For the reason behind the 50-rep requirement for commenting, please read meta.stackexchange.com/a/214174/231617. $\endgroup$ – Waldir Leoncio Jul 10 '16 at 11:38
  • $\begingroup$ With only a little adaptation / augmentation, this could be your own answer, instead of just a comment on another answer. Then you would be able to start earning the reputation necessary to comment. I urge you to do so. $\endgroup$ – gung - Reinstate Monica Jul 10 '16 at 12:03
  • $\begingroup$ Thanks guys, I have edited the answer, and removed reference to the previous reply, I hope it might be useful for someone, and I have also realised that I have another question of my own on this subject, but will post that separately. Sorry for taking myself liberties with respect to the community rules. $\endgroup$ – cjrberg Jul 10 '16 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.