Bootstrapping where each sample size is the same size as the whole sample?

Question

I am going through an online course on Data Analysis and Statistical Inference, and I am having trouble understanding the Bootstrapping method. I am asking the question here, as the course is not very well developed.

In the course, we repeatedly take samples from a variable "weight gained" (973 clean data points), but the size of each sample is the size of the length of the variable (i.e. taking all 973 data points each time), we calculate the mean for each sample, and then 'replace' the sample for the next batch.

So I guess my question is, why would the mean of each sample be different, if I am taking all the data points each time? I must be missing something here.

The R code we are using is:

# Initialize the 'boot_means' object:
boot_means = rep(NA, 100)

# Insert your for loop:
for(i in 1:100){
  boot_sample = sample(gained_clean, n, replace = TRUE)
  boot_means[i] = mean(boot_sample)
}

# Make a histogram of 'boot_means':
hist(boot_means)

Answer 1

(Note that you can make a reproducible example by setting the seed and generating some data. This facilitates people working on your problem and helping them help you, because both can reproduce the same results.)

Did you try it? Did you notice that you actually get back a vector of identical means? Let's try it here:

set.seed(7793)
n = 973
gained_clean = rnorm(n, mean=0, sd=1)
summary(gained_clean)
#     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
# -3.79300 -0.69380 -0.03424 -0.05384  0.59740  3.34300 
confint(lm(gained_clean~1), level=.90)
#                    5 %         95 %
# (Intercept) -0.1066353 -0.001050521

boot_means = rep(NA, 100)

# Insert your for loop:
for(i in 1:100){
  boot_sample = sample(gained_clean, n, replace = TRUE)
  boot_means[i] = mean(boot_sample)
}
summary(boot_means)
#      Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
# -0.124800 -0.070680 -0.052570 -0.054840 -0.033300  0.006434 
sort(boot_means)[5]   # [1] -0.1121571
sort(boot_means)[95]  # [1] -0.009469124

The first thing to notice is that the population from which the data were generated has a mean of 0. However, the realized data have a mean of -0.05384. Nonetheless, because this is a simulation, we pretend we don't know the true data generating process and take our sample as a model of the population itself. The bootstrap gives us a sense of what the sampling distribution of the mean from our sample-as-population would look like. Notice that the mean of that sampling distribution (-0.054840) is centered on the sample mean. Furthermore, note that the 90% confidence interval derived from the bootstrapped sampling distribution is [-0.112, -0.009]. In this case, it doesn't cover the true mean, but in the long run it will 90% of the time and is quite similar to the analytically derived confidence interval [-0.107, -0.001].

So (getting to your explicit question), how did this work? The key is that bootstrapping (and the R function ?sample(x, size, replace=TRUE)) is not returning your exact sample each time. Instead, it returns a sample drawn with replacement. Let's re-run the above code (especially setting the seed again for reproducibility), but this time taking a look at the first few data:

...  # some code skipped
head(gained_clean)
# [1]  0.12336275  0.42359922 -0.09915415 -0.05301904  0.51948019  0.20804275
boot_samples_matrix = matrix(rep(NA, n*5), nrow=n)
for(i in 1:5){
  boot_samples_matrix[,i] = sample(gained_clean, n, replace = TRUE)
}
cbind(head(gained_clean), head(boot_samples_matrix))
#             [,1]       [,2]       [,3]        [,4]       [,5]       [,6]
# [1,]  0.12336275  0.6149658 -0.3477429  0.04203624  0.4091023 -0.4760131
# [2,]  0.42359922 -0.2879273  0.7248926 -0.83440292 -0.3450141 -0.3688194
# [3,] -0.09915415  0.2208385  0.4497207 -0.45045587  0.1422502 -0.3079226
# [4,] -0.05301904 -0.9771884 -1.0765040  0.50582315 -0.9665563  1.9123963
# [5,]  0.51948019 -0.4365715 -0.9910226 -0.23829268  0.2485701 -1.7783639
# [6,]  0.20804275  1.6725675  1.1873863 -0.22584317 -0.2014681  0.3144087

Now we can see that they aren't the same each time at all. Of course, these could have been just shuffled, so let's check for that:

sum(duplicated(gained_clean))  # [1] 0
apply(boot_samples_matrix, MARGIN=2, FUN=function(x){ sum(duplicated(x)) })
# [1] 343 370 343 366 362

There were no duplicated data in the original vector gained_clean, but there were about 357, on average, duplicated data in each boot sample.