Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

Filter by
Sorted by
Tagged with
0
votes
0answers
31 views

R Caret - Random Forest with repeated CV

I am currently struggling to explain the combination of bagging and cross-validation. I am aware of what each does separately but I have difficulties to explain how they are combined (if at all). For ...
3
votes
1answer
226 views

Is sampling with replacement better than sampling without replacement?

This question is more specific to machine learning. Is sampling with replacement good for random forests because it leaves some out of bag samples for testing or is it because it creates datasets/...
0
votes
0answers
34 views

RandomForest with VST data

I used the randomForest() function in R for bagging without a training data set to identify the important features characterizing the members of groups A and B (binary testing). I get the following ...
6
votes
1answer
40 views

Why is the size of the feature bagging sample typically the square root of the total predictor set size?

Applying feature sampling to each tree in the context of a Random Forest model, if the set of predictors is of size $p$, why does the size of the sample of predictors for each split typically be of ...
0
votes
1answer
19 views

Can we make bagging equivalent to random forest if we take max features in our own defined decision tree and pass it as estimator?

If in bagging we defined our own tuned decision tree as estimator with max features as parameter in that estimator rather than taking by default decision tree estimator available in BaggingClassifer ...
7
votes
2answers
105 views

Intutition of why Bootstrap aggregating reduces overfitting?

Can somebody give me a non-mathematical intuition why Bootstrap aggregating reduces overfitting? From my point of view, we are not providing any additional information, we are not really enlengthen ...
0
votes
0answers
11 views

Why does my Bagging Ensemble model overfit? [duplicate]

I implemented a Bagging Ensemble model in sklearn, but the cross_validate method always returns a perfect train score. And here are my results Is this behaviour normal or am I doing something wrong ...
0
votes
0answers
15 views

Boosting models performing worse than bagging models

I've noticed that, sometimes, models that are based on boosting, such as Gradient Boosting, show worse performance than pretty similar models, but based on bagging. For example, Random Forest (bagging ...
1
vote
1answer
20 views

Why can't we sample without replacement for each tree in a random forest if the subsample size is large enough?

Usually if we have $n$ observations, for each tree with form a bootstrapped subsample of size $n$ with replacement. On googling it one common explanation I've seen is that with replacement sampling is ...
0
votes
0answers
15 views

Proof that bagging increases bias (estimation of squared mean)

We have independent and identically distributed training data $\mathcal{T} = \{z_{i}\}_{i=1}^{N}$, where $z_{i} = ( y_{i}, \mathbf{x}_{i})$, $y_{i} \in\mathbb{R}$. Further, suppose we want to estimate ...
1
vote
2answers
265 views

AdaBoost - why decision stumps instead of trees?

Since the original AdaBoost article it has been found out that boosting reduces both variance and bias in the classifier (in contrast to bagging, which only reduces variance). Original AdaBoost (and ...
2
votes
2answers
41 views

Elements of Statistical Learning: Variance Reduction via Bagging for Random Forests

While Reading chapter 15 of ESL about Random Forests I had some confusion with this phrase: The idea in random forests (Algorithm 15.1) is to improve the variance reduction of bagging by reducing the ...
0
votes
1answer
32 views

Is there a model, apart from Random Forest, which uses bagging?

What are the popular machine learning algorithms which make use of bagging? Are there any apart from random forest?
1
vote
0answers
28 views

Regarding bagging, boosting and the NFL theorem

According to my understanding, bagging and boosting work in the following way: Bagging: Combine several high-variance/low-bias models to produce an ensemble model with lower variance and equal bias. ...
0
votes
1answer
26 views

Do tree methods like gradient boosting predict all iterations at once?

If I'm using a tree method (e.g GBM) and I have a time series hourly data, and I predict my target variable $y$ for the next 48 hours, do my predictions were made all at once, or does the second day ...
0
votes
1answer
40 views

How does bagging effect linear model assumptions?

Linear regression has model assumptions. How does bagging effect model assumptions for linear regression? Also, should you build a bagged linear model with correlated and statistically significant ...
2
votes
0answers
69 views

Performing Bagging with the Lasso

I asked this question at StackOverflow but I think it might be better suited for here because I'm hoping to understand the general idea of bagging. I'm trying to implement bagging in ...
0
votes
1answer
55 views

bias-variance tradeoff is conflict to bagging

I first give the notations of bias-variance tradeoff: sample set: $D = \{(x_1,y_1),\cdots,(x_N,y_N)\}.$ relation between $y$ and $x:$ $y = f(x) + \epsilon.$ here $f(x)$ is a deterministic function; $...
3
votes
2answers
129 views

How many neglected samples when drawn with replacement? (bagging)

I learned a while ago about an interesting place that $e$ shows up in probability: if there are $n$ items and you sample $n$ times with replacement, you would expect that the fraction of samples that ...
1
vote
0answers
11 views

Is there a way to know which feature is causing a split or present in root or decision node of a decision tree in Random Forest using Python? [closed]

Let's say I have 100 features f1, f2, .... ,f100 and I want to know if f3, f4, f5 caused any split or were present in a root/decision node of any decision tree in a random forest. I have close to 200 ...
0
votes
0answers
37 views

Stacking of Machine Learning models using different data splits

I have one single dataset with 2 classes. I want to make a model for binary classification, and I am experimenting a bit. My intention is to use stacking on some models by using subsets of the 1 ...
1
vote
1answer
140 views

What is the limit of the random forest (or bagging) estimator?

I am looking for a proof or intuition as to why the absolute limit of a random forest estimator is the expectancy of a single tree (see citation below), i.e: $$ \hat{f}_{rf}(x) = \lim_{B \to \infty}[\...
1
vote
0answers
17 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
2
votes
1answer
122 views

Is n_estimators in BaggingRegressor() the number of trees or data subsets?

I'm learning about using the BaggingRegressor() from scikit learn (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html) Its <...
2
votes
1answer
398 views

How does bagging reduce variance?

I read this answer. Was still unable to understand how bagging reduces variance. Is there any other way to explain it mathematically to a newbie ? Edit Can anybody explain me this excerpt from the ...
1
vote
1answer
68 views

Bagging models with different metaparameters versus cross validation?

If we have a model with a metaparameter C, the usual way to tune this parameter is via cross-validation (or building a validation set). The generalization error of the model with the optimal is ...
2
votes
1answer
165 views

Overfit in aggregated models: boosting versus simple bagging

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the ...
3
votes
1answer
286 views

What is the purpose of using duplicated data in resampling techniques (e.g., bagging/bootstrapping)?

With bootstrapping and bagging, we resample from the dataset and end up building a model or estimating some sample statistic using the sampled data, which typically consists of at least $33\%$ ...
1
vote
1answer
21 views

Some ambiguities about used training dataset in bagged trees

I have some ambiguity about dividing training dataset in Bagging tree. In fact I have found in this article About Decision Tree Ensembles- Bagging That : the idea is to create several subsets of data ...
1
vote
2answers
75 views

Is boosting and bagging only relevant in the context of decision trees?

In the documents I've seen on boosting and bagging, it seems that they're always talked about in the context of decision trees. What are some other methods in which the two are applicable?
0
votes
0answers
27 views

Are predictions generated from bootstrap resampling normally distributed?

If I resample a dataset hundreds of times with replacement (bootstrap replications), fit some Model A on each bootstrap sample, and generate predictions (time series) from each fitted model, can I ...
3
votes
2answers
194 views

Bagging - Size of the aggregate bags?

I'm reading up on bagging (boostrap aggregation), and several sources seem to state that the size of the bags (consist of random sampling from our training set with replacement) is typically around 63%...
2
votes
1answer
17 views

Bagging sample probability

I know that the probability that an observation does not appear in the sample is $\left(1 - \frac{\ 1}{n} \right)^n ≈ \left(\frac{1}{e}\right) $ (when n is large). My question is how to find the ...
0
votes
0answers
530 views

Time complexity of bagging and random forest

Most sources and books state that time complexity of a single decision tree for n points and d dimensions (features) is $O(d * n^2 * log(n))$, with clever caching and one time sorting it’s $O(d * n * ...
2
votes
1answer
122 views

Why are boosted trees difficult to interpret?

I might not fully understand the topic but: Bootstrapped/bagged trees are difficult to interpret because the decisions are made from averaging the prediction of possibly hundreds of trees (ensemble). ...
2
votes
0answers
10 views

Random forest and bagging: if both generate deep, non-pruned trees then what's the difference?

I understand that random forest only considers a subset of predictors at each node, but if you are generating deep trees then you'll eventually include all predictors in your model, the only ...
1
vote
1answer
130 views

Decision tree-based Bagging and Random Forest

I am new to Machine Learning and would like to know is it always true that decision tree-based bagging has worse predictive performance and is slower running time than random forest?
1
vote
0answers
31 views

Why do correlated decision trees increase bias?

To decrease variance in decision trees often, bagging is used to create several decision trees. This has better accuracy than one tree since the decrease in variance outweighs the increase in bias. ...
2
votes
1answer
118 views

Why is the error rate from bagging trees much higher than that from a single tree?

I'm running the classification method Bagging Tree (Bootstrap Aggregation) and compare the misclassification error rate with one from one single tree. We expect that the result from bagging tree is ...
0
votes
0answers
36 views

What is the meaning of component err.rate of class randomForest?

I asked this question on Stackoverflow, but it's likely that I will received no answer on that site. So I cross-post my question here. I'm using the function ...
4
votes
1answer
2k views

Why do we use random sample with replacement while implementing random forest?

Let's say we want to build random forest. Wikipedia says that we use random sample with replacement to do bagging. I don't understand why we can't use random sample without replacement.
0
votes
1answer
142 views

Bagging results from multiple k-fold cross-validation runs to estimate performance

Given a binary classification problem and a model construction algorithm, suppose I run, as an example, 5 fold (stratified) cross-validation 4 times. I can think of these as 5x4 = 20 training and 20 ...
1
vote
0answers
38 views

Trying to improve the generalization ability of decision tree with bagging or random forest

We are fitting a regression tree on a sample of about 5000 observations with 5 predictors. The data stems from 2 sources. There are reasons to suspect that the tree will not have exaggerated ...
0
votes
1answer
53 views

Machine Learning - Bagging

I am very new to Machine Learning I know i should use Decision Tree for bagging model because of its high variance. why cant we use any other algorithms for bagging ? if we can use any other ...
3
votes
1answer
595 views

Why is bagging stable classifiers not a good idea?

Citing Bagging Predictors, Section 4 (emphasis mine): Let each $(y, \mathbf{x})$ case in $\mathcal{L}$ be independently drawn from the distribution $P$. Suppose $y$ is numerical and $\phi(\mathbf{x}...
1
vote
1answer
535 views

why does random forest trees need to be deeper than gradient boosting trees

in Elements of Statistical Learning chapter 15. Random Forest, we see authors' note on RF v.s. GBT. One of them is that at 1000 terms, GBM depth 4 has smaller error than RF depth 6. Also we notice RF ...
0
votes
2answers
79 views

Using bagging and random forests together

I was looking at a kernel implementation (for text classification) and the following piece of code got me a little bit confused (I removed part of the features - in order to keep it light - as most of ...
3
votes
1answer
981 views
1
vote
0answers
18 views

Bagging Classifiers

In Bootstrap aggregating, instances that have not been drawn (by resampling with replacement), e.g. end up for example in the test set. However, in the bootstrap sample not every instance is unique. ...
1
vote
0answers
58 views

Does it make sense to use bagging with least square regression? if so, why?

We know that bagging can reduce the variance of the estimator, while keep the bias around the same. If we use bagging to ensemble multiple least square regressors, then are we going to reduce the ...