Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

Filter by
Sorted by
Tagged with
276
votes
8answers
196k views

Bagging, boosting and stacking in machine learning

What's the similarities and differences between these 3 methods: Bagging, Boosting, Stacking? Which is the best one? And why? Can you give me an example for each?
55
votes
6answers
48k views

Is random forest a boosting algorithm?

Short definition of boosting: Can a set of weak learners create a single strong learner? A weak learner is defined to be a classifier which is only slightly correlated with the true ...
35
votes
2answers
5k views

Is this the state of art regression methodology?

I've been following Kaggle competitions for a long time and I come to realize that many winning strategies involve using at least one of the "big threes": bagging, boosting and stacking. For ...
22
votes
2answers
32k views

What does “node size” refer to in the Random Forest?

I do not understand exactly what is meant by node size. I know what a decision node is, but not what node size is.
22
votes
1answer
3k views

Boosting AND Bagging Trees (XGBoost, LightGBM)

There are many blog posts, YouTube videos, etc. about the ideas of bagging or boosting trees. My general understanding is that the pseudo code for each is: Bagging: Take N random samples of x% of ...
20
votes
3answers
4k views

When should I not use an ensemble classifier?

In general, in a classification problem where the goal is to accurately predict out-of-sample class membership, when should I not to use an ensemble classifier? This question is closely related to ...
19
votes
1answer
2k views

What are the theoretical guarantees of bagging

I've (approximately) heard that: bagging is a technique to reduce the variance of an predictor/estimator/learning algorithm. However, I have never seen a formal mathematical proof of this ...
18
votes
5answers
13k views

Are Random Forest and Boosting parametric or non-parametric?

By reading the excellent Statistical modeling: The two cultures (Breiman 2001), we can seize all the difference between traditional statistical models (e.g., linear regression) and machine learning ...
17
votes
3answers
10k views

Why does a bagged tree / random forest tree have higher bias than a single decision tree?

If we consider a full grown decision tree (i.e. an unpruned decision tree) it has high variance and low bias. Bagging and Random Forests use these high variance models and aggregate them in order to ...
16
votes
2answers
8k views

Why not always use ensemble learning?

It seems to me that ensemble learning WILL always give better predictive performance than with just a single learning hypothesis. So, why don't we use them all the time? My guess is because of ...
16
votes
2answers
10k views

Why does the scikit-learn bootstrap function resample the test set?

When using bootstrapping for model evaluation, I always thought the out-of-bag samples were directly used as a test set. However, this appears not to be the case for the deprecated scikit-learn ...
15
votes
1answer
457 views

What bagging algorithms are worthy successors to Random Forest?

For boosting algorithms, I would say that they evolved pretty well. In early 1995 AdaBoost was introduced, then after some time it was Gradient Boosting Machine (GBM). Recently, around 2015 XGBoost ...
14
votes
7answers
3k views

Random Forest and Decision Tree Algorithm

A random forest is a collection of decision trees following the bagging concept. When we move from one decision tree to the next decision tree then how does the information learned by last decision ...
10
votes
1answer
10k views

Random Forest Probabilistic Prediction vs majority vote

Scikit learn seems to use probabilistic prediction instead of majority vote for the model aggregation technique without an explanation as to why (1.9.2.1. Random Forests). Is there a clear ...
10
votes
1answer
790 views

Why does bagging use bootstrap samples?

Bagging is the process of creating N learners on N different bootstrap samples, then taking the mean of their predictions. My question is: Why not use any other type of sampling? Why use bootstrap ...
10
votes
1answer
2k views

Should pruning be avoided for bagging (with decision trees)?

I came by several posts and papers claiming that pruning trees in a "bagging" ensemble of trees is not needed (see 1). However, is it necessarily (or at least in some known cases) damaging to perform ...
9
votes
1answer
5k views

How can we explain the fact that “Bagging reduces the variance while retaining the bias” mathematically?

I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias". What is the mathematically principle behind this intution? I checked with few experts ...
9
votes
1answer
3k views

Confusion related to the bagging technique

I am having a bit of confusion. I was reading this paper where it explained that bagging technique greatly reduces variance and only slightly increases bias. I didn't get it how come it reduces ...
9
votes
1answer
2k views

Can I combine many gradient boosting trees using bagging technique

Based on Gradient Boosting Tree vs Random Forest . GBDT and RF using different strategy to tackle bias and variance. My question is that can I resample dataset (with replacement) to train multiple ...
8
votes
1answer
3k views

Is it pointless to use Bagging with nearest neighbor classifiers?

In page 485 of the book [1], it is noted that "it is pointless to bag nearest-neighbor classifiers because their output changes very little if the training data is perturbed by sampling". This is ...
8
votes
1answer
4k views

Bagging of xgboost

The extreme-gradient boosting algorithm seems to be widely applied these days. I often have the feeling that boosted models tend to overfit. I know that there are parameters in the algorithm to ...
7
votes
1answer
2k views

Main idea of Bagging

I just read this post and several other websites, but I still don't understand what bagging is. I understand it is an algorithm for machine learning, that it improves stability and accuracy of the ...
7
votes
1answer
821 views

Variance of average of $n$ correlated random variables

Reading about deep leaning, I came across the following formula. $$ \mbox{var} \left( \frac{1}{n} \sum_{i=1}^{n} X_i \right) = \rho \sigma^2 + \frac{1-\rho}{n} \sigma^2 $$ where $X_1, \dots, X_n$ ...
7
votes
1answer
4k views

Why is tree correlation a problem when working with bagging?

I'm reading ISLR and I don't understand what is the problem that random forests solve; what problems does tree correlation cause when using bagging?
7
votes
2answers
24k views

Is cross validation unnecessary for Random Forest?

Is it fair to say Cross Validation (k-fold or otherwise) is unnecessary for Random Forest? I've read that is the case because we can look at out-of-bag performance metrics, and these are doing the ...
7
votes
1answer
3k views

Scalable Random Forest For Massive Data

My problem is simple. I want to train a dataset using random forest on a huge dataset (with $n$ rows). Let's assume I can only fit $b < n$ rows in memory at a time. Model Choice I see several ...
6
votes
2answers
8k views

How is bagging different from cross-validation?

How is bagging different from cross-validation? Can a data set having 300 examples can be 100 bagged and would it be helpful at all?
6
votes
1answer
2k views

Where must we use Bagging or Boosting?

I want to know when Bagging is better than Boosting? How I select appropriate method for my classification task? I think when we have many outliers in our data-set, Bagging must be better than ...
6
votes
1answer
2k views

Why does bagging increase bias?

In machine learning, why does bagging increase bias? I've read that using less data would lead to a worse estimate of the parameters, but isn't the expected value of the parameter constant regardless ...
6
votes
1answer
6k views

XGBoost feature subsampling

I have a dataset with ~30k samples and 35 features (after feature selection; these seem to be the most important features for this dataset and they have low correlation between each other). After ...
5
votes
2answers
5k views

Are K-Fold Cross Validation , Bootstrap ,Out of Bag fundamentally same?

Can Anyone tell me how K-Fold Cross Validation ,Bootstrap and Out of Bag Approach differ as they use 1)Separate data into training data and testing data 2)Make model using training data and ...
5
votes
3answers
1k views

Meaning of Bagged Random Forests?

I'm reading a paper that says that the authors used "bagged random forests". I couldn't understand this because as far as I know a random forest is a kind of bagging on its own. So a random forest is ...
5
votes
2answers
546 views

Ensembles of Ensembles?

I like the idea of ensemble learners, especially Bagging, but I always wondered as why they are not the most powerful learners since they have a clean motivation. I don't have the answer to that ...
5
votes
1answer
3k views

Bootstrap aggregation (bagging) of logistic regression classifiers

So I'm taking N bootstrap samples and training N logistic regression classifiers on these samples. Each classifier gives me some probability of being in a binary class and then I average these N ...
5
votes
0answers
357 views

Is there an appropriate order to apply bagging and filter feature selection?

I'm training a (regression) learner on a $p \gg n$ problem, including bagging and filter feature selection (information gain). I'm in doubt though regarding the order of the procedures: Apply the ...
4
votes
2answers
4k views

When should the Pasting ensemble method be used instead of Bagging?

Pasting and Bagging are very similar, the main difference being that Bagging samples with replacement (which is called "bootstrapping") while Pasting samples without replacement. I am guessing that ...
4
votes
1answer
2k views

Coefficients of bootstrapping in logistic regression

I have seen several articles and CrossValidated questions on bootstrapping ( this, this or this for example); there are a lot of theoretical and statistical explanations, however since they are so ...
4
votes
1answer
1k views

Why do we use random sample with replacement while implementing random forest?

Let's say we want to build random forest. Wikipedia says that we use random sample with replacement to do bagging. I don't understand why we can't use random sample without replacement.
4
votes
2answers
2k views

What are advantages of random forests vs using bagging with other classifiers?

I'm studying Random Forests, but after reviewing the methods I got the following line of reasoning: I feel like the big advantage of random forests happens in the bagging process where nearly ...
4
votes
0answers
358 views

Bag of Features / Visual Words + Locality Sensitive Hashing

PREMISE: I'm really new to Computer Vision/Image Processing and Machine Learning (luckily, I'm more expert on Information retrieval), so please be kind with this filthy peasant! :D MY APPLICATION: ...
4
votes
0answers
89 views

Is bagging a free lunch w.r.t. generalization error?

Since bagging seems to reduce variance without increasing the size of the hypothesis set (I think), is it fair to say that it does not increase the bias? Therefore, in terms of out-of-sample error, it ...
4
votes
2answers
658 views

Bagging vs pasting: bias-variance tradeoff

In the Hands-On ML with Scikit-Learn book, it states that, ...bagging ends up with a slightly higher bias than pasting, but... the ensemble's variance is reduced. I am a bit confused about this ...
3
votes
1answer
2k views

Why does Bagging improve accuracy?

I am reading about the idea of bagging (boostrap aggregating). I have no trouble understanding how the variance of prediction can be reduced. The simple picture is if you have prediction Z_1 through ...
3
votes
3answers
1k views

What base classifiers to use with bagging?

I am working on a specific implementation of bootstrap aggregating (bagging). I want to see how well this bagging works for different base classifiers. But so far, the decision tree seems to be the ...
3
votes
1answer
2k views

what is the effect of bootstap resampling in bagging algorithm(ensemble learning)?

In ensemble learning with bagging, why is it important to do bootstrap resampling (sampling with replacement) instead of just sub-sampling (sampling without replacement)?
3
votes
1answer
526 views

Why is bagging stable classifiers not a good idea?

Citing Bagging Predictors, Section 4 (emphasis mine): Let each $(y, \mathbf{x})$ case in $\mathcal{L}$ be independently drawn from the distribution $P$. Suppose $y$ is numerical and $\phi(\mathbf{x}...
3
votes
1answer
851 views
3
votes
1answer
258 views

Theoretical Justifications for Random Forest

Is there any theoretical justifications for Random Forests in high dimensions? I notice the work "Uniform Convergence of Random Forests via Adaptive Concentration" which shows generalization of RF ...
3
votes
1answer
182 views

What is the purpose of using duplicated data in resampling techniques (e.g., bagging/bootstrapping)?

With bootstrapping and bagging, we resample from the dataset and end up building a model or estimating some sample statistic using the sampled data, which typically consists of at least $33\%$ ...
3
votes
2answers
142 views

Bagging - Size of the aggregate bags?

I'm reading up on bagging (boostrap aggregation), and several sources seem to state that the size of the bags (consist of random sampling from our training set with replacement) is typically around 63%...