Bagging vs pasting: bias-variance tradeoff

Question

In the Hands-On ML with Scikit-Learn book, it states that,

...bagging ends up with a slightly higher bias than pasting, but... the ensemble's variance is reduced.

I am a bit confused about this part. Wouldn't bagging have higher variance and lower bias, since the sampled instances will be more correlated with each other compared to pasting? (Similar to how leave-one-out CV has higher variance due to higher correlation compared to K-fold.)

Or, is it just because bagging can sample more instances and train higher number of predictors compared to pasting? But in this case, bagging will have lower variance but not necessarily higher bias?

Answer 1

I wasn't totally sure about the pasting method. From Hands-On Machine Learning with Scikit-Learn it's written :

Another Approach is to use the same training algorithm for every predictor, but to train them on different random subsets of the training set. When sampling is perform with replacement, this method is called bagging, when sampling is performed without replacement it is called pasting

Because the bagging method use replacement, you can see data that wouldn't appear IRL ( let say two occurences of the same unique data), so the bias can be higher.

But, you are not limited by the number of classifier you train, because you can take the same data other times. The more classifier you train, the less variance you have

Answer 2

After thinking for a long time, my conclusion is that the author is not referring to the MSR version of variance but this on page 126:

The Bias/Variance Tradeoff
--------------------------

An important theoretical result of statistics and Machine Learning is the fact that a
model’s generalization error can be expressed as the sum of three very different
errors:

**Bias**

This part of the generalization error is due to wrong assumptions, such as assuming
that the data is linear when it is actually quadratic. A high-bias model is most
likely to underfit the training data.10

**Variance**

This part is due to the model’s excessive sensitivity to small variations in the
training data. A model with many degrees of freedom (such as a high-degree polynomial
model) is likely to have high variance, and thus to overfit the training
data.

To answer your question more specifically:

When we use bagging (with replacement), we are more likely to learn from the same training instances and have an underfit result in the end. Thus, the model would be less sensitive to small variances in the training data (since the chances are less likely to be selected by a particular predictor).