While studying machine learning I read about different sampling methods. Simple holdout, N-fold cross validation are straightforward. However, I somehow miss the point of bootstrapping. Its definition says that it's just a way to inflate the sample set simply by duplicating some random samples and I cannot figure what is the point in this -- seemingly no additional information in a learning process just by seeing the same instances again and again (on the contrary: others say that omitting redundant points from the training set is recommended for computational efficiency and for some other statistical reason as well).

So what is the explanation here?

  • $\begingroup$ But you're not seeing the same samples again and again. You're taking a different sample each time. With bootstrapping, you're taking a simple random sample with replacement from the original sample. You end up with many different samples. For example, if your original sample was $S={1,2,3}$ some bootstrap samples might be: $S_1^*={1,1,1}$, $S_2^*={1,2,3}$, $S_3^*={3,2,3}$, $S_2^*={2,2,3}$, etc. $\endgroup$ Dec 11, 2020 at 8:30
  • $\begingroup$ But in case of S1 = 1, 1, 1 I still show my estimator the same 1 instance three times. It's pointless. $\endgroup$
    – Fredrik
    Dec 11, 2020 at 8:50
  • 1
    $\begingroup$ If you want to sample from a distribution defined in terms of samples, redundant samples are equivalent to changing the weighs, or importance, of those redundant samples. $\endgroup$
    – crlb
    Dec 11, 2020 at 8:55
  • $\begingroup$ I see, but in bootstrapping redundant samples are choosen randomly, which in this sense means that we assign weights randomly for certain points. Why, and on what basis? This also seems irrational. $\endgroup$
    – Fredrik
    Dec 11, 2020 at 9:03
  • $\begingroup$ The distribution $S_1 = \{a,a,b\}$ with equal probability for all elements and the distribution $S_2 = \{a,b\}$ with $p_a = 2/3$ and $p_b = 1/3$ are equivalent when bootstrapping. $\endgroup$
    – crlb
    Dec 11, 2020 at 9:08


Browse other questions tagged or ask your own question.