2
$\begingroup$

Sometimes in a paper I see that (e.g.) confidence intervals were estimated using 99 (or 199 or 999) bootstraps. My question is, why 99 rather than 100?

This is my best guess at an answer:

"If we sort the results from 100 bootstraps, the 5th and 95th results are estimates of the 5th and 95th percentiles of the whole population, which would give us a 90% confidence interval. But the estimator only converges as sample size goes to infinity. With a finite sample, the estimator will be inaccurate, and half the time we will get a too narrow confidence interval. Taking 99 bootstraps, the 5th and 95th results are conservative estimates of the population percentiles. For reasonable sample sizes, they are pretty sure to cover the true interval between 5th and 95th population percentiles."

Is that right, or close?

$\endgroup$
1

1 Answer 1

2
$\begingroup$

Obviously, it doesn't matter that much, bootstrap works the same with 99 or 100 samples. There are a few reasons why someone might choose to do it that way.

  1. You perform 99 bootstrap repetitions but add your real model as 100th, so in the end, you will get a nice round number
  2. 99 bootstrapped results give you 99 percentiles that divide your number line to 100 buckets (as when you have one median, that gives you two halves, or two tertiles that give you three thirds, etc.), so in the end, you will get a nice round number
  3. Somewhere in the denominator is +1, probably to make the results a little bit more realistic/conservative so that you won't get p-value == 0, which shouldn't really happen

I am doing a similar thing with a permutation test when we run 999 permutations because the last 1000th is considered to be the real observed result so that the smallest possible p-value depends on the number of permutations and can never be 0

$\endgroup$
1
  • $\begingroup$ I think the correct answer is given in the comment above, not here. $\endgroup$
    – dash2
    Commented Oct 31, 2022 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.