5
$\begingroup$

With bootstrapping and bagging, we resample from the dataset and end up building a model or estimating some sample statistic using the sampled data, which typically consists of at least $33\%$ duplicate data.

My questions are:

(1) Why do even need to use the duplicated data? Why don't we simply discard them, and just use the unique data for each bag?

(2) When the duplicated data is used, aren't you putting more emphasis on those data during the learning process? If so, doesn't that introduce some bias into the model?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ The gist of bootstrap is that you're treating the discrete, empirical distribution as a population. When you sample from a discrete distribution, you might get duplicated values. $\endgroup$
    – Dave
    Commented Sep 3, 2020 at 18:23
  • 5
    $\begingroup$ There's a similar question and answer here. I like to think of a duplicated draw as representing another member of the underlying population that has values close to the case that's being duplicated from your original sample. $\endgroup$
    – EdM
    Commented Sep 3, 2020 at 21:07
  • $\begingroup$ @EdM Just read your answer in that post. So essentially, the idea is if we're allowed to sample with replacement, as with bootstrapping, then each draw from the pool is independent? But then this seems to come with the cost that the samples within the bootstrapped dataset is not entirely independent. If you resampled without replacement, then your bootstrapped dataset samples would be independent (assuming the original pool of data was independently generated)? $\endgroup$
    – 24n8
    Commented Sep 3, 2020 at 23:15
  • $\begingroup$ Having identical values doesn't make samples dependent. Think about rolling a die repeatedly: there are only 6 possible outcomes so you will get many values repeated as you roll over and over. Nevertheless, the distribution among those values can provide information about whether the die is fair or not. In bootstrapping, the data that you have are considered to be a discrete representation of the underlying population. Sampling without replacement then means that subsequent samples depend on what happened in prior samples. That's dependency in sampling even if all values are distinct. $\endgroup$
    – EdM
    Commented Sep 4, 2020 at 14:54

1 Answer 1

Reset to default
2
$\begingroup$

Pretty sure the statistical properties of bootstrap hold only for when n of your bootstrap sample equals the N of your actual sample so duplicates are necessary. This is pretty easy to see with some simulation of the sampling process in general, sampling without replacement and some arbitrary percentage of your N gives you, on average, more biased results.

Bagging when it comes to predictors don't require N = n but as an approach it is very useful for increasing the bias and decreasing the variance when it comes to things such as trees. Bagging linear models will just converge to a normal linear model but you can use those estimates to do analysis on rather than standard formulas which is a fun thing to do.

You are putting more weight on the duplicates for that specific draw but you are drawing hundreds or thousands of times.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.