When should the Pasting ensemble method be used instead of Bagging?

Question

Pasting and Bagging are very similar, the main difference being that Bagging samples with replacement (which is called "bootstrapping") while Pasting samples without replacement.

I am guessing that bootstrapping introduces more randomness, so Bagging may result in a slightly higher bias, but the estimators in the ensemble will be less correlated, which will reduce the ensemble's variance, hopefully making better predictions?

Is this guess correct? When would you use one versus the other? Would you simply always use cross-validation to evaluate both and see which one works best?

Answer 1

I am not an expert on the subject, but I think I have a sufficient answer:

Since pasting is without replacement, each subset of the sample can be used once at most, which means that you need a big dataset for it to work. As a matter of fact, pasting was originally designed for large data-sets, when computing power is limited. Bagging, on the other hand, can use the same subsets many times, which is great for smaller sample sizes, in which it improves robustness (to my experience).

So, I think size is the major factor for making this decision. If your sample size is small, pasting isn't a real option. When it is, I would expect bagging to yield better cross-validation results almost always, but pasting might prove better in external validations (i.e. real life predictions), as it reaches its conclusion by aggregating predictions from practically independent datasets.

Answer 2

Bagging is to use the same training for every predictor, but to train them on different random subsets of the training set. When sampling is performed with replacement, this method is called bagging (short for bootstrap aggregating). When sampling is performed without replacement, it is called pasting. In other words, both approaches are similar. In both cases you are sampling the training data to build multiple instances of a classifier. In both cases a training item could be sampled and used to train multiple instances in the collection of classifiers that is produced. In bagging, it is possible for a training sample to be sampled multiple times in the training for the same predictor. This type of bootstrap aggregation is a type of data enhancement, and it is used in other contexts as well in ML to artificially increase the size of the training set.

Computationally bagging and pasting are very attractive because in theory and in practice all of the classifiers can be trained in parallel. Thus if you have a large number of CPU cores, or even a distributed memory computing cluster, you can independently train the individual classifiers all in parallel.

scikit-learn

using scikit-learn for performing bagging and/or pasting is relatively simple. As with the voting classifier, we specify which type of classifer we want to use. But since bagging/pasting train multiple classifiers all of this type, we only have to specify 1. The n_jobs parameter tells scikit-learn the number of cpu cores to use for training and predictions (-1 tells scikit-learn to use all available cores).

The following trains an ensemble of 500 decision tree classifiers (n_estimators), each trained on 100 training instances randomly sampled from the training set with replacement (bootstrap=True). If you want to use pasting we simply set bootstrap=False instead.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

bagging_clf = BaggingClassifier(
    DecisionTreeClassifier(max_leaf_nodes=20), n_estimators=500,
    max_samples=100, bootstrap=True, n_jobs=-1
)

bagging_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)