Two ways of using bootstrap to estimate the confidence interval of coefficients in regression

Question

I am applying a linear model to my data: $$ y_{i}=\beta_{0}+\beta_{1}x_{i}+\epsilon_{i}, \quad\epsilon_{i} \sim N(0,\sigma^{2}). $$

I would like to estimate the confidence interval (CI) of the coefficients ($\beta_{0}$, $\beta_{1}$) using bootstrap method. There are two ways that I can apply the bootstrap method:

1. Sample paired response-predictor: Randomly resample pairs of $y_{i}-x_{i}$, and apply linear regression to each run. After $m$ runs, we obtain a collection of estimated coefficients ${\hat{\beta_{j}}}, j=1,...m$. Finally, compute the quantile of ${\hat{\beta_{j}}}$.

2. Sample error: First apply linear regression on the original observed data, from this model we obtain $\hat{\beta_{o}}$ and the error $\epsilon_{i}$. Afterwards, randomly resample the error $\epsilon^{*}_{i}$ and compute the new data with $\hat{\beta_{o}}$ and $y^{*}_{i}=\hat{\beta_{o}}x_{i}+\epsilon^{*}_{i}$. Apply once again linear regression. After $m$ runs, we obtain a collection of estimated coefficeints ${\hat{\beta_{j}}}, j=1,...,m$. Finally, compute the quantile of ${\hat{\beta_{j}}}$.

My questions are:

• How are these two methods different?
• Under which assumption are these two methods giving the same result?

Answer 1

If the response-predictor pairs have been obtained from a population by random sample, it is safe to use case/random-x/your-first resampling scheme. If predictors were controlled for, or the values of the predictors were set by the experimenter, you may consider using residual/model-based/fixed-x/your-second resampling scheme.

How do the two differ? An introduction to the bootstrap with applications in R by Davison and Kounen has a discussion pertinent to this question (see p.9). See also the R code in this appendix by John Fox, particularly functions boot.huber on p.5 for the random-x scheme and boot.huber.fixed on p.10 for the fixed-x scheme. While in the lecture notes by Shalizi the two schemes are applied to different datasets/problems, Fox's appendix illustrate how little difference the two schemes may often make.

When can the two be expected to deliver near identical results? One situation is when the regression model is correctly specified, e.g., there is no unmodelled nonlinearity and the usual regression assumptions (e.g., iid errors, no outliers) are satisfied. See chapter 21 of Fox's book (in which the aforementioned appendix with the R code indirectly belongs), particularly the discussion on page 598 and exercise 21.3. entitled "Random versus fixed resampling in regression". To quote from the book

By randomly reattaching resampled residuals to fitted values, the [fixed-x/model-based]
procedure implicitly assumes that the errors are identically distributed. If, for
example, the true errors have non-constant variance, then this property will not be  
reflected in the resampled residuals. Likewise, the unique impact of a high-leverage
outlier will be lost to the resampling.

You will also learn from that discussion why fixed-x bootstrap implicitly assumes that the functional form of the model is correct (even though no assumption is made about the shape of the error distribution).

See also slide 12 of this talk for Society Of Actuaries in Ireland by Derek Bain. It also has an illustration of what should be considered "the same result":

The approach of re-sampling cases to generate pseudo data is the more usual form of   
bootstrapping. The approach is robust in that if an incorrect model is fitted an
appropriate measure of parameter meter uncertainty is still obtained. However re
sampling residuals is more efficient if the correct model has been fitted.

The graphs shows both approaches in estimating the variance of a 26 point data sample
mean and a 52 point sample mean. In the larger sample the two approaches are  
equivalent.