# Two ways of using bootstrap to estimate the confidence interval of coefficients in regression

I am applying a linear model to my data: $$y_{i}=\beta_{0}+\beta_{1}x_{i}+\epsilon_{i}, \quad\epsilon_{i} \sim N(0,\sigma^{2}).$$

I would like to estimate the confidence interval (CI) of the coefficients ($\beta_{0}$, $\beta_{1}$) using bootstrap method. There are two ways that I can apply the bootstrap method:

1. Sample paired response-predictor: Randomly resample pairs of $y_{i}-x_{i}$, and apply linear regression to each run. After $m$ runs, we obtain a collection of estimated coefficients ${\hat{\beta_{j}}}, j=1,...m$. Finally, compute the quantile of ${\hat{\beta_{j}}}$.

2. Sample error: First apply linear regression on the original observed data, from this model we obtain $\hat{\beta_{o}}$ and the error $\epsilon_{i}$. Afterwards, randomly resample the error $\epsilon^{*}_{i}$ and compute the new data with $\hat{\beta_{o}}$ and $y^{*}_{i}=\hat{\beta_{o}}x_{i}+\epsilon^{*}_{i}$. Apply once again linear regression. After $m$ runs, we obtain a collection of estimated coefficeints ${\hat{\beta_{j}}}, j=1,...,m$. Finally, compute the quantile of ${\hat{\beta_{j}}}$.

My questions are:

• How are these two methods different?
• Under which assumption are these two methods giving the same result?
• I would personally not use either as the default approach but instead would recommend the basic bootstrap confidence interval. See p. 8 of www.stat.cmu.edu/~cshalizi/402/lectures/08-bootstrap/lecture-08.pdf‎ . I've been doing a lot of simulations for the binary logistic model and have seen better confidence interval coverage using the basic bootstrap than using the percentile or BCa bootstrap. – Frank Harrell Jul 19 '13 at 11:26
• @FrankHarrell to be clear, by "basic" you are referring to the non-parametric bootstrap? – ndoogan Jul 19 '13 at 13:08
• (1) is the bootstrap percentile nonparametric confidence interval, not the basic bootstrap. Note that sampling from $(x,y)$ is the unconditional bootstrap, which is more assumption-free than the conditional bootstrap that resamples residuals. – Frank Harrell Jul 19 '13 at 14:59
• I'm really not an expert, but as far as I understand it, 1) is often called "case-resampling" whereas the 2) is called "residual resampling" or "fixed-$x$" resampling. The basic choice of the method doesn't imply the method of how to calculate the confidence intervals after the procedure. I got this info mainly from the tutorial of John Fox. As far as I see it, after either bootstrap, you could calculate the basic bootstrap CIs (e.g. with boot.ci(my.boot, type="basic") in R). Or do I miss anything here? – COOLSerdash Jul 19 '13 at 16:23
• Yes, you can do cluster bootstrapping. This is implemented in the R rms validate and calibrate functions. – Frank Harrell Jul 19 '13 at 20:38

If the response-predictor pairs have been obtained from a population by random sample, it is safe to use case/random-x/your-first resampling scheme. If predictors were controlled for, or the values of the predictors were set by the experimenter, you may consider using residual/model-based/fixed-x/your-second resampling scheme.

How do the two differ? An introduction to the bootstrap with applications in R by Davison and Kounen has a discussion pertinent to this question (see p.9). See also the R code in this appendix by John Fox, particularly functions boot.huber on p.5 for the random-x scheme and boot.huber.fixed on p.10 for the fixed-x scheme. While in the lecture notes by Shalizi the two schemes are applied to different datasets/problems, Fox's appendix illustrate how little difference the two schemes may often make.

When can the two be expected to deliver near identical results? One situation is when the regression model is correctly specified, e.g., there is no unmodelled nonlinearity and the usual regression assumptions (e.g., iid errors, no outliers) are satisfied. See chapter 21 of Fox's book (in which the aforementioned appendix with the R code indirectly belongs), particularly the discussion on page 598 and exercise 21.3. entitled "Random versus ﬁxed resampling in regression". To quote from the book

By randomly reattaching resampled residuals to ﬁtted values, the [fixed-x/model-based]
procedure implicitly assumes that the errors are identically distributed. If, for
example, the true errors have non-constant variance, then this property will not be
reﬂected in the resampled residuals. Likewise, the unique impact of a high-leverage
outlier will be lost to the resampling.


You will also learn from that discussion why fixed-x bootstrap implicitly assumes that the functional form of the model is correct (even though no assumption is made about the shape of the error distribution).

See also slide 12 of this talk for Society Of Actuaries in Ireland by Derek Bain. It also has an illustration of what should be considered "the same result":

The approach of re-sampling cases to generate pseudo data is the more usual form of
bootstrapping. The approach is robust in that if an incorrect model is fitted an
appropriate measure of parameter meter uncertainty is still obtained. However re
sampling residuals is more efficient if the correct model has been fitted.

The graphs shows both approaches in estimating the variance of a 26 point data sample
mean and a 52 point sample mean. In the larger sample the two approaches are
equivalent.