3
$\begingroup$

I think I have a handle on what bootstrapping is and why we need to use it.

Please confirm if my understanding is correct:

Goal of bootstrapping: To find the SE of a feature’s coefficient that you estimated in your ML model.

Background: We have an Original dataset of n = 3. We ran a ML model on it and came up with a coef value for B1. This coef value is just a guess as of now. How do we determine how “correct” is this coef value?

Bootstrapping Steps:

  1. Randomly select 10 observations (w/ replacement) from original dataset. Lets call this Bootstrapped Dataset #1.

  2. Calculate B1 from Bootstrapped Dataset #1.

  3. Repeat steps 1 and 2 100 times.

  4. Now we have 100 B1’s from Bootstrapped Datasets #1-100. Calc the SE of those 100 B1’s. This represents the SE of B1 in the Original dataset.


Yes I inserted numbers like 10 and 100 to make things simpler and easier to understand vs. mathematical notation. I understand that this would be done n times in the real world.

$\endgroup$
4
  • 3
    $\begingroup$ If n=3, why are you selecting 10 observations? Bootstrap resamples are the same size as the original data, so if n is the sample size, you would draw samples of size 3. $\endgroup$
    – Sycorax
    Commented Jan 11, 2023 at 2:32
  • 1
    $\begingroup$ The original purpose of the jackknife, precursor to the bootstrap, was to reduce the bias in a biased estimator. The estimation of the se was secondary. $\endgroup$ Commented Jan 11, 2023 at 3:50
  • $\begingroup$ Try this article arxiv.org/abs/1411.5279 $\endgroup$ Commented Jan 11, 2023 at 5:46
  • 1
    $\begingroup$ Please read stats.stackexchange.com/questions/26088/…. $\endgroup$
    – whuber
    Commented Jan 11, 2023 at 13:31

1 Answer 1

6
$\begingroup$

You're on the right track, but some of the details are a bit off.

A common goal of statistics to make inferences about large populations. For example, we might want to know how many people are expected to vote for each candidate, or whether children taught one way learn more than those taught a different way. It's often infeasible or impossible to completely survey the entire population, so we instead rely on smaller samples drawn from that population. However, there's a catch: sampling introduces variability, because one subset of the population likely differs, if only by chance, from another. To make claims about the populations, we want to understand how much variability is attributable to the sampling itself, rather than differences between populations.

Bootstrapping provides a way to do that by treating your sample as a small population and drawing new samples from it (this is why it is called a "resampling" method). If your original observations are independent and identically distributed, the process works pretty much as you say. (More complicated methods are needed if data points are correlated, like in a time series).

  1. Draw a new sample, with replacement, from your original sample. It should be the same size as the original. For example, if your original sample had 100 observations, the new sample should contain 100 as well, though some data points are likely going to be duplicated (due to the replacement), while others may be missing. If you only have three observations in total, you're not going to be able to do much (with any technique!).

  2. Compute the statistic of interest on that new sample and record it. This could be the mean, but it doesn't necessarily have to be: you can bootstrap variances, correlation coefficients, and almost anything else. Often, your statistic will be an estimator (like the sample mean) of a population parameter (like the population mean).

  3. Repeat this process a large number of times.

The result is the sampling distribution of your statistic. As you say, this is often collapsed down to a single value, like a standard error or confidence interval, though you don't necessarily have to do. However, the standard error is computed by taking the standard deviation of the sampling distribution (not the standard error, as your question said). Critically, the standard error doesn't tell you anything about correctness. What it does tell you is about the sampling variability in your statistic: how much would (say) the mean change if you had drawn a different sample from the same population.

Finally, you didn't quite touch on why we use the bootstrap. Closed-form or asymptotic solutions for sampling variation exist, but they often make/require certain assumptions. The bootstrap largely sidesteps this problem.

$\endgroup$
1
  • 3
    $\begingroup$ Perfect answer. I wish all statistics textbooks were written like this. $\endgroup$
    – Katsu
    Commented Jan 11, 2023 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.