1. Bootstrap before calculating CIs
Not sure if I understood your question correctly, but if you were asking, ‘Should I be using bootstrap to compute the CIs’, then, the missing part of the question is, ‘bootstrap instead of what’, ‘more precise’ – ‘precise compared to what and in which sense’.
There are multiple ways to construct the CIs:
- Make an assumption about the sampling distribution of the estimator the for every sample size $n$. This is a very strong assumption; however, people have done it many times in the past: they often assume the $t$ distribution (for sample sizes at least 2). This is the classical frequentist (Fisher) inference.
- Assume that nothing is known anything about the sampling distribution for any given $n$, but as $n \to\infty$, you know the distribution: $n^q (\hat \theta_n - \theta_0) \xrightarrow[n\to\infty]{d} \mathcal{N}(0, V)$, where $q$ is the rate of convergence ($q=1/2$ for most parametric estimators, $q = 1/5$ or lower for non-parametric ones, $q=2/5$ for the smoothed Manski estimator etc.). Then, you just look up the table of Gaussian critical values. The problem is, estimating $V$ is sometimes non-trivial.
- Estimate the critical value by bootstrapping. This is where the consistency of bootstrap is required (i.e. no parameter on the boundary of the parameter space, the same rate of convergence of the original and bootstrap estimators etc. – in general, the failure of $\sup_{u\in \mathbb{R}} |\mathrm{CDF}_{\sqrt{n}(\hat\theta^*_n - \hat\theta_n)} (u) - \mathrm{CDF}_{\sqrt{n}(\hat\theta_n - \theta_0)} (u)| \xrightarrow[n\to\infty]{\mathbb{P}} 0$ (in Efron’s notation), which can happen due to a multitude of reasons).
So if bootstrap works in the sense of the (rather technical) condition described above, then, depending on some extra conditions (such as requiring finite estimator variances), it can beat the asymptotic confidence intervals (as well as the variance estimators, $p$-values – basically, any functional of the estimator distribution) in the sense of the approximation error. Assume that $\mathbb{E} (\hat\theta_n - \theta_0)^2 < \infty$ (which rules out certain estimators, like the IV estimator that is the ratio of two Gaussians) and that the sampling distribution of $\hat\theta_n$ is symmetrical. Then, bootstrap is ‘better’ in the following sense:
$$\sup_{u\in \mathbb{R}} |\mathrm{CDF}_{\sqrt{n}(\hat\theta_n - \hat\theta_0) / \mathrm{SE} \hat\theta_n} (u) - \Phi(u)| = O(1/\sqrt{n}),$$
$$\sup_{u\in \mathbb{R}} |\mathrm{CDF}_{\sqrt{n}(\hat\theta_n^* - \hat\theta_n) / \mathrm{SE}^* \hat\theta^*_n} (u) - \mathrm{CDF}_{\sqrt{n}(\hat\theta_n - \theta_0) / \mathrm{SE} \hat\theta_n} (u)| = O_p(1/n),$$
where $\Phi$ is the CDF of the standard normal distribution.
Or course it does not guarantee that the capital O in a specific given application is not going to bring the refinement, and of course, depending on the smoothness of the bootstrapped quantity (bias, or variance, or CI, or p-value) and the bootstrap type, the refinement may or may not exist – however, if you are worrying that the bootstrap is going to be less reliable than asymptotic confidence intervals – probably not. Bootstrap does a much better job on reproducing the shape of the sampling distribution (on which one should really be doing inference), which is why if your method belongs to a broad class of estimators for which bootstrap works and you can afford running the bootstrap sufficiently many times, then, bootstrap should be more trustworthy in the sense of capturing the extra moments of the sampling distribution uncertainty (as opposed to the first 2 moments of the asymptotic Gaussian approximation). Depending on how poorly the Weak Law of Large Numbers is working when the numbers are not large enough, bootstrap may simply unveil it to the researcher.
NB. Bootstrap is an asymptotic method in the sense that it still relies in most aspects on the number of observations $n \to \infty$. Bootstrap does not improve theoretical properties of statistical tests, and if $n=8$, the question would be, ‘is scientific method really applicable?’ or ‘aren’t we making conclusions about random data features, not the true underlying relationships?’. If the theoretical power of one’s test is low or there are extra complications in the form of departures from the ‘randomised controlled trial’ setting (as well as ‘independent identically distributed’), bootstrap won’t help, and the paper will be rejected.
2.1 Number of replications (practical advice)
A large number of replications B is required to say that the finite-sample Monte-Carlo approximation replicates the asymptotic (in B) bootstrap distribution of the object of interest closely enough. This means that there is no penalty (other than increased computation time) to doing more replications in one experimental setting. Unless one is studying the theoretical properties of bootstrap (e.g. nested bootstrap, second-order bootstrap, bootstrapping new estimators etc.), then, the infallible rule is ‘more is better’.
In case one does not have deep bootstrap knowledge, here is a quick recommendation (that should hopefully stay relevant for another decade).
- B >= 1000, otherwise your paper will be rejected with something like ‘We are not in the Pentium-II era’ from Referee 2.
- Ideally, B >= 10000; try to do it if your computer can handle it.
Here is where most researchers may stop. However, if the researcher suspects that their sampling distribution may be irregular and discrepancies between the true and simulated distribution are large, then, we may check some features of the bootstrap distribution to determine how close we are to it (as a function of $B$). The seminal paper is Andrews & Buchinsky (2000, Econometrica). Here are the extra steps to make any picky referee shut up:
- You could check if your B yields the desired probability $1-\tau$ of achieving the desired relative accuracy $r$ of the bootstrapped quantity of interest for some common level (e.g. $r= 5\%$ and $\tau=5\%$).
- If not, increase B to the value dictated by the A&B 3-stage procedure described below.
- In general, for any actual accuracy of your bootstrapped quantity, to increase the desired relative accuracy by a factor of k, increase B by a factor of $k^2$.
2.2 A data-driven theory-backed procedure
There is a data-driven method of choosing B: do some small number of bootstrap replications, see how stable or noisy the estimator is, and then, based on some target accuracy measure, increase the number of replications until you are sure that this resampling-related error has reached a certain lower bound with a chosen certainty. Our helper here is the Weak Law of Large Numbers where the asymptotics are in B. To be more specific, B is chosen depending on the user-chosen bound on the relative deviation measure of the Monte-Carlo approximation of the quantity of interest based on B simulations. This quantity can be standard error, p-value, confidence interval, or bias correction. The closeness is the relative deviation $R^*$ of the B-replication bootstrap quantity from the infinite-replication quantity (or, to be more precise, the one that requires $n^n$ replications): $R^* := (\hat\lambda_B - \hat\lambda_\infty)/\hat\lambda_\infty$. The idea is, find such B that the actual relative deviation of the statistic of interest be less than a chosen bound (usually 5%, 10%, 15%) with a specified high probability $1-\tau$ (usually $\tau = 5\%$ or $10\%$). Then,
$$\sqrt{B} \cdot R^* \xrightarrow{d} \mathcal{N}(0, \omega),$$
where $\omega$ can be estimated using a relatively small (usually 200–300) preliminary bootstrap sample that one should be doing in any case.
Here is the general formula for the number of necessary bootstrap replications $B$:
$$B \ge \omega \cdot (Q_{\mathcal{N}(0, 1)}(1-\tau/2) / r)^2,$$
where r is the maximum allowed relative discrepancy (i.e. accuracy), $1-\tau$ is the probability that this desired relative accuracy bound has been achieved, $Q_{\mathcal{N}(0, 1)}$ is the quantile function of the standard Gaussian distribution, and $\omega$ is the asymptotic variance of $R$*. The only unknown quantity here is $\omega$ that represents the variance due to simulation randomness.
The general 3-step procedure for choosing B is like this:
- Compute the approximate preliminary number $B_1 := \lceil \omega_1 (Q_{\mathcal{N}(0, 1)}(1-\tau/2) / r)^2 \rceil$, where $\omega_1$ is a very simple theoretical formula from Table III in Andrews & Buchinsky (2000, Econometrica).
- Using these $B_1$ samples, compute an improved estimate $\hat\omega_{B_1}$ using a formula from Table IV (ibid.).
- With this $\hat\omega_{B_1}$ compute $B_2 := \lceil\hat\omega_{B_1} (Q_{\mathcal{N}(0, 1)}(1-\tau/2) / r)^2 \rceil$ and take $B_{\mathrm{opt}} := \max(B_1, B_2)$.
If necessary, this procedure can be iterated to improve the estimate of $\omega$, but this 3-step procedure as it is tends to yield already conservative estimates that ensure that the desired accuracy has been achieved. This approach can be vulgarised by taking some fixed $B_1 = 1000$, doing 1000 bootstrap replications in any case, and then, doing steps 2 and 3 to compute $\hat\omega_{B_1}$ and $B_2$.
Example (Table V, ibid.): to compute a bootstrap 95% CI for the linear regression coefficients, in most practical settings, to be 90% sure that the relative CI length discrepancy does not exceed 10%, 700 replications are sufficient in half of the cases, and to be 95% sure, 850 replications. However, requiring a smaller relative error (5%) increases B to 2000 for $\tau=10\%$ and to 2700 for $\tau=5\%$.
This agrees with the formula for B above. If one seeks to reduce the relative discrepancy r, by a factor of k, the optimal B goes up roughly by a factor of $k^2$, whilst increasing the confidence level that the desired closeness is reached merely changes the critical value of the standard normal (1.96 → 2.57 for 95% → 99% confidence).
