Can you explain bootstrapping like I’m 5? I think I have a handle on what bootstrapping is and why we need to use it.
Please confirm if my understanding is correct:
Goal of bootstrapping: To find the SE of a feature’s coefficient that you estimated in your ML model.
Background: We have an Original dataset of n = 3. We ran a ML model on it and came up with a coef value for B1. This coef value is just a guess as of now. How do we determine how “correct” is this coef value?
Bootstrapping Steps:

*

*Randomly select 10 observations (w/ replacement) from original dataset. Lets call this Bootstrapped Dataset #1.


*Calculate B1 from Bootstrapped Dataset #1.


*Repeat steps 1 and 2 100 times.


*Now we have 100 B1’s from Bootstrapped Datasets #1-100. Calc the SE of those 100 B1’s. This represents the SE of B1 in the Original dataset.

Yes I inserted numbers like 10 and 100 to make things simpler and easier to understand vs. mathematical notation. I understand that this would be done n times in the real world.
 A: You're on the right track, but some of the details are a bit off.
A common goal of statistics to make inferences about large populations. For example, we might want to know how many people are expected to vote for each candidate, or whether children taught one way learn more than those taught a different way. It's often infeasible or impossible to completely survey the entire population, so we instead rely on smaller samples drawn from that population. However, there's a catch: sampling introduces variability, because one subset of the population likely differs, if only by chance, from another. To make claims about the populations, we want to understand how much variability is attributable to the sampling itself, rather than differences between populations.
Bootstrapping provides a way to do that by treating your sample as a small population and drawing new samples from it (this is why it is called a "resampling" method). If your original observations are independent and identically distributed, the process works pretty much as you say. (More complicated methods are needed if data points are correlated, like in a time series).

*

*Draw a new sample, with replacement, from your original sample. It should be the same size as the original. For example, if your original sample had 100 observations, the new sample should contain 100 as well, though some data points are likely going to be duplicated (due to the replacement), while others may be missing. If you only have three observations in total, you're not going to be able to do much (with any technique!).


*Compute the statistic of interest on that new sample and record it. This could be the mean, but it doesn't necessarily have to be: you can bootstrap variances, correlation coefficients, and almost anything else. Often, your statistic will be an estimator (like the sample mean) of a population parameter (like the population mean).


*Repeat this process a large number of times.
The result is the sampling distribution of your statistic. As you say, this is often collapsed down to a single value, like a standard error or confidence interval, though you don't necessarily have to do. However, the standard error is computed by taking the standard deviation of the sampling distribution (not the standard error, as your question said). Critically, the standard error doesn't tell you anything about correctness. What it does tell you is about the sampling variability in your statistic: how much would (say) the mean change if you had drawn a different sample from the same population.
Finally, you didn't quite touch on why we use the bootstrap. Closed-form or asymptotic solutions for sampling variation exist, but they often make/require certain assumptions. The bootstrap largely sidesteps this problem.
