# Can we not use Bootstrapping for all small sample ML use cases?

This might be a noob question and excuse me for my naivette here.

But we know in Statistics, we can use Bootstrapping to generate more samples repeatedly from the same sample.

So on problems where we are dealing with small samples (less observations), can't we use bootstrapping and do ML then?

It can be for Regression or Classification problem.

• One does not typically use the bootstrap to generate a larger sample for statistical inference. Generally, one uses it to infer the sampling distribution of a statistic (eg, the mean) given the sample you have observed. Nov 5, 2019 at 18:48
• Can you clarify what the goal of this would be? As aocall mentioned, this is not why bootstrapping is used, and explaining why might be easier if we understood more about why you want to do this. Nov 5, 2019 at 18:53
• Well, I have been asked questions from ppl that when I am doing an ML like Regression, the no. of observations were very small and I should do Bootstrapping. I am not sure that is right, but wanted to check if Bootstrapping small obs to create a larger sample and then do ML is right? Nov 5, 2019 at 19:35
• It is more than not typical to use the bootstrap to generate a larger sample, it is wrong. The bootstrap resamples are used under conditions where it is valid to approximate the sampling distribution for statistics of interest as @aocall points out as well as other applications. Nov 5, 2019 at 19:40
• Bootstrapping can be applied in small samples just like other parametric & nonparametric procedures. But bootstrapping cannot be used as though you have a larger sample. The bootstrap can be justified when a parametric family of distributions cannot be credibly used. An important thing to remember about bootstrapping is that it does not add any information than what is available in the original sample. Nov 5, 2019 at 19:46

Perhaps it is useful to separate two parameters here:

1. the parameter of interest (say, the population mean), and
2. the standard error of our estimator of the parameter of interest.

Bootstrapping is generally used to assist with (2). One use case is when we don't have an analytical expression for the standard error of our estimator. Another is when we do but our analytical expression is based on an asymptotic approximation. We might worry that the asymptotic approximation is not close enough to the finite-sample behavior of our estimator. To quell our worry, under certain conditions the bootstrap can provide an asymptotic refinement.

The bootstrap can't lend more statistical power to the estimator for (1). Put differently, we don't use it to "generate" a larger sample for (1). We use it to simulate the sampling distribution and estimate (2).

• I am looking at this example. here a small sample was used and then bootstrapping applied to it and an ML model was fit. My question is does Bootstrapping help u build better ML models on small sample or is it basically useful to get uncertainity around the ML model skill (estimate/accuracy) on test data? machinelearningmastery.com/… Nov 5, 2019 at 20:18
• So u r saying.. if I have a dataset of 20 obs.. and I want to build a regression model on it. Then by doing bootstrapping, I can't build a better ML model (resampling those 10 obs and fit ML). It will only help me find a distribution of some statistic (like accuracy rmse score etc) and find CI of that. The model built on those 10 obs without Bootstrap will be same performance as that built with doing Bootstrap? Nov 5, 2019 at 20:22
• One way to think about it is this. Suppose you have a sample of $(X_i)_{i=1}^{20}$. $\hat\mu \equiv \sum_i X_i / 20$ estimates the population mean with variance $Var(X_i)/20$. An alternative estimator uses each observation twice (effectively doubling the sample by duplicating every observation): $\tilde\mu \equiv \sum_i 2 X_i / 40 = \sum_i X_i / 20 = \hat\mu$. The variance of $\tilde\mu$ is therefore the same so we did not gain precision. Nov 6, 2019 at 0:01
• This is a good comment. But then how does Random Forest and ensemble methods are able to get a better statistical learning model on bootstrapped data from a sample data than a single decision tree built on the same sample data without bootstrapping? Does it hv to do with ensemble (majority vote) more than bootstrapping? Nov 6, 2019 at 1:11