How does bootstrap sampling work in practice?

Question

Could someone please describe, using the reproducible code sample below, how bootstrap sampling works in practice? The more detailed questions are:

• What different random datasets are generated from myData by bootstrap sampling it?

• How myData looks like after the sampling with replacement has been done?

Thanks!

> x <- c(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)

> y <- 2*x

> myData <- data.frame(x, y)

> library(ipred)

> myModel <- ipred::bagging(y ~ ., data = myData, nbagg = 3, ns = length(y))

nbagg = the number of bootstrap replications

ns = the relative number of samples (from all training data) used for boostrapping

Answer 1

You have already figured out the basics in your own answer.

To specifically address your questions (and make this an on-topic programming question, and not an off-topic theoretical/tutorial one), i.e.

What different random datasets are generated from myData by bootstrap sampling it?

How myData looks like after the sampling replacement has been done?

change your code slightly to

set.seed(42) # for reprocucibility
myModel <- ipredbagg(y, x, nbagg = 3, ns = length(y))

after which, you can see the specific bootstrapped samples used for each one of your nbagg=3 individual classifiers:

> myModel$mtrees[[1]]$bindx
 [1] 10 10  3  9  7  6  8  2  7  8  5  8
> myModel$mtrees[[2]]$bindx
 [1] 10  3  5 10 10  2  5  6 10  2 10 10
> myModel$mtrees[[3]]$bindx
 [1]  1  6  4 10  5  9  8  9  4  7  1  9

keeping in mind that the above are the indices of your original x, and not the actual data (bindx stands for 'bootsrap index'); to get the data themselves, you can use

> x[myModel$mtrees[[1]]$bindx]
 [1] 9 9 2 8 6 5 7 1 6 7 4 7
> x[myModel$mtrees[[2]]$bindx]
 [1] 9 2 4 9 9 1 4 5 9 1 9 9
> x[myModel$mtrees[[3]]$bindx]
 [1] 0 5 3 9 4 8 7 8 3 6 0 8

and similarly for y.

Answer 2

After some investigations and a deleted answer to my question I figured it out. Bootstrap sampling with replacement means that a data subset is created from the original data set by selecting samples that can be also the same ones as previously selected. So, there is a chance that some samples in a data subset are different but some are the same.

In my case (the code presented in my question), the original data set is myData consists of 10 samples. In the function ipred::bagging parameters, nbagg = 3 and ns = length(y) mean that three different models and three data subsets (let's say A, B and C) of full size (= the amount of samples in myData) for these models will be created. Because of bootstrap sampling with replacement, the data subset A will consist of 10 samples (= length(y) i.e. 10) picked from myData randomly such that the same sample can occur more than once, e.g.: [1 2], [2 4], [5 10], [1 2] (<- repeated sample), ..., [2 4] (<- repeated sample), [8 16] (<- the final i.e. 10th sample).