Linear Model Residual Bootstrapping

I am having some issues how to figure out what the second iteration of a fixed-x resampling bootstrap would be. For instance if I have this data

Y   x1  x2
7   2   3
6   3   5
5   4   1
9   5   4


and I calculate the coefficients

        Coefficients
Intcp   3.137931034
x1      0.551724138
x2      0.517241379


Getting the following residuals:

1.206896552
-1.379310345
-0.862068966
1.034482759


On my first bootstrap iteration I sample the residuals and add those samples residuals to the Y from the previous model. I sampled -0.862068966, -1.379310345, -0.862068966, 1.206896552 in this case. Getting the following table:

Y           x1  x2
6.137931034 2   3
4.620689655 3   5
4.137931034 4   1
10.20689655 5   4



Coefficients
Intercept   -0.096313912
x1          1.235434007
x2          0.63020214


and residuals of

1.206896552
-1.379310345
-0.862068966
1.034482759


For the next bootstrap, am I supposed to continue this chain and sample the last set of residuals and add them to the last set of Ys? Or do I use either the original Y values or original residuals in anyway?

In bootstrapping, you repeatedly sample with replacement from the original sample. You can simply add this to your loop:

wh <- sample(1:nrow(original.sample), replace = TRUE)
boot.sample <- original.sample[wh, ]
boot.lm <- lm(Y ~ x1 + x2, data = boot.sample)


For fixed-x, or residual resampling as you intend, you sample with replacement from the original residuals in each iteration.

residuals <- resid(model)
wh <- sample(1:length(residuals), replace = TRUE)
boot.Y  <- original.Y + residuals[wh]
boot.lm <- lm(boot.Y ~ x1 + x2)

• Just to clarify, for the next iteration of bootstrap, this means I would sample with replacement from the original residuals (1.2, -1.37, -0.86, 1.03) and add them to the Y values calculated in the previous bootstrap (6.13, 4.6, ...), correct?
– Alex
Commented Oct 11, 2017 at 3:13
• Bootstrapping means to sample with replacement from the original sample $(y_i, x_i)$, as shown in the example code. Could you clarify why you insist on sampling the residuals? Commented Oct 11, 2017 at 3:18
• I am specifically trying to create a bootstrap model using fixed-x resampling. Section 3.2 of this document is what I am trying to replicate
– Alex
Commented Oct 11, 2017 at 3:26
• Ah, Googling 'fixed x resampling' clarified that, I was unfamiliar with this technique. Perhaps you could include it in the question. To answer your comment: Yes, you resample from the original residuals in each step. Commented Oct 11, 2017 at 3:29
• I have updated my answer to include this method Commented Oct 11, 2017 at 3:32