# Understanding the Bootstrap method in *Introduction to Statistical Learning*

I am having a hard time understanding Bootstrap method. In the book Introduction to Statistical Learning (pp. 187-190) the Bootstrap method is explained by first using "simulated datasets" from an "original population" to estimate $$\alpha$$. But this approach does not seem to be applicable in practical scenarios, so we use Bootstrap samples instead. The book says that Bootstrap samples are taken from the "original data set".

I really don't understand these terms mean. Could someone please explain to me in simple words what those pages mean by these terms, and hence, what does this mean about how the Bootstrap method works?

Bootstrapping is introduced as a method to estimate the variance of a statistics $$S$$, given a sample $$X=\{X_1, X_2, \ldots, X_2\}$$.

Usually, you can have two different scenarios. One scenario is when you know the analytical expression of the distribution of $$S$$. One when you cannot infer the distribution of $$S$$. In the first case, you can easily use the theoretical results to calculate the property of $$S$$. In the second case, bootstrapping can provide a workaround to give you an approximated form of the distribution of $$S$$.

The book gives an example, where $$X$$ and $$Y$$ are random variables and you are interested in the random variable $$\alpha$$ which is a function of the variance of $$X$$ and $$Y$$, $$\sigma_X^2$$, $$\sigma_Y^2$$, and their covariance $$\sigma_{XY}$$. Since these are unknown, you can estimate them from the sample, getting $$\hat{\alpha}$$.

Here, they simulate different scenarios to show you what the variability of $$\hat{\alpha}$$ can be.

Remember that in reality, you have only one $$\hat{\alpha}$$.

But, let's say that you know the expected distributions of $$X$$ and $$Y$$, then you can simulate different scenarios, by sampling $$X$$ and $$Y$$ from the expected distributions.
For instance, let's say that $$Z=(X, Y)$$ is a 2-dimensional random variable defined by merging $$X$$ and $$Y$$. Let's suppose that $$Z$$ is distributed as a 2-dimensional Normal distribution with mean $$\mu_Z=(\mu_X, \mu_Y)$$, and variance-covariance matrix $$\Sigma$$.

$$Z|\mu_Z, \Sigma \sim N(\mu_Z, \Sigma)\\ \Sigma = \pmatrix{\sigma_X^2 & \sigma_{XY} \\ \sigma_{XY} & \sigma_Y^2}$$

Then, simulating 1000 times this stochastic system is equivalent to randomly sample $$Z$$ from the given distribution.
This is a R snippet where you need to specify the means and variance, covariance values:

mu_z = (mu_x, mu_y)
cov_mat = rbind(c(sig_x, sig_xy), c(sig_xy, sig_y))

z = MASS::mvrnorm(n=1000, mu=mu_z, Sigma=cov_mat)


Now that you have 1000 $$Z$$ ($$X$$, $$Y$$ pairs), you can calculate $$\hat{\alpha}$$ from each.
The advantage of simulating is that we know the theoretical variances and covariance from which we sampled the $$X$$ and $$Y$$, so we can compare the simulated $$\hat{\alpha}$$ with the "true" $$\alpha$$.

As you can see, they show that the maximum likelihood estimation of $$\alpha$$ (its mean) is pretty close to the theoretical value 0.6

$$\bar{\alpha}=\frac{1}{1000}\sum_{i=1}^{1000} \hat{\alpha}_i=0.5996$$

with a standard deviation equal 0.083. This value tells you how close the estimate is to the real $$\alpha$$.

Now we go back to the real scenario, where you have only one sample with unknown distribution.

The question is how can we estimate the error of our estimate with only one sample?

This is where bootstrap comes into play.

Bootstrapping is a procedure to estimate the variability of a random variable given a single sample.

The procedure is simple. You repeat a large number of times $$i=1,\ldots,N$$:

1. Randomly sample $$n$$ observations ($$n$$ equal to the number of observations in the dataset) from the dataset with replacement
2. Estimate your statistics ($$\hat{\alpha_i}$$ in our example) using this sample

Finally:

1. Calculate mean and standard deviation or quantile intervals

Now, you can realise the principle behind this procedure.
The main assumption is that your sample is one realisation of infinite possible scenarios all distributed following some theoretical distribution. That means that there is an unknown distribution driving the phenomenon, and when you observe the phenomenon you are randomly sampling from that distribution. Parameters of this distribution are fixed (unknown), data vary.

When you don't have multiple samples, you can assume that your data is roughly representing the statistical properties of the underlying population. Then you can randomly sample with replacement and estimate the variability of your statistics.