Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

Filter by
Sorted by
Tagged with
2
votes
2answers
27 views

Elements of Statistical Learning: Variance Reduction via Bagging for Random Forests

While Reading chapter 15 of ESL about Random Forests I had some confusion with this phrase: The idea in random forests (Algorithm 15.1) is to improve the variance reduction of bagging by reducing the ...
0
votes
0answers
17 views

Is there a model, apart from Random Forest, which uses bagging?

What are the popular machine learning algorithms which make use of bagging? Are there any apart from random forest?
1
vote
0answers
21 views

Regarding bagging, boosting and the NFL theorem

According to my understanding, bagging and boosting work in the following way: Bagging: Combine several high-variance/low-bias models to produce an ensemble model with lower variance and equal bias. ...
1
vote
1answer
25 views

Do tree methods like gradient boosting predict all iterations at once?

If I'm using a tree method (e.g GBM) and I have a time series hourly data, and I predict my target variable $y$ for the next 48 hours, do my predictions were made all at once, or does the second day ...
0
votes
1answer
19 views

How does bagging effect linear model assumptions?

Linear regression has model assumptions. How does bagging effect model assumptions for linear regression? Also, should you build a bagged linear model with correlated and statistically significant ...
2
votes
0answers
39 views

Performing Bagging with the Lasso

I asked this question at StackOverflow but I think it might be better suited for here because I'm hoping to understand the general idea of bagging. I'm trying to implement bagging in ...
0
votes
1answer
44 views

bias-variance tradeoff is conflict to bagging

I first give the notations of bias-variance tradeoff: sample set: $D = \{(x_1,y_1),\cdots,(x_N,y_N)\}.$ relation between $y$ and $x:$ $y = f(x) + \epsilon.$ here $f(x)$ is a deterministic function; $...
3
votes
2answers
91 views

How many neglected samples when drawn with replacement? (bagging)

I learned a while ago about an interesting place that $e$ shows up in probability: if there are $n$ items and you sample $n$ times with replacement, you would expect that the fraction of samples that ...
1
vote
0answers
10 views

Is there a way to know which feature is causing a split or present in root or decision node of a decision tree in Random Forest using Python? [closed]

Let's say I have 100 features f1, f2, .... ,f100 and I want to know if f3, f4, f5 caused any split or were present in a root/decision node of any decision tree in a random forest. I have close to 200 ...
0
votes
0answers
18 views

Stacking of Machine Learning models using different data splits

I have one single dataset with 2 classes. I want to make a model for binary classification, and I am experimenting a bit. My intention is to use stacking on some models by using subsets of the 1 ...
1
vote
1answer
67 views

What is the limit of the random forest (or bagging) estimator?

I am looking for a proof or intuition as to why the absolute limit of a random forest estimator is the expectancy of a single tree (see citation below), i.e: $$ \hat{f}_{rf}(x) = \lim_{B \to \infty}[\...
1
vote
0answers
17 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
2
votes
1answer
45 views

Is n_estimators in BaggingRegressor() the number of trees or data subsets?

I'm learning about using the BaggingRegressor() from scikit learn (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html) Its <...
2
votes
1answer
193 views

How does bagging reduce variance?

I read this answer. Was still unable to understand how bagging reduces variance. Is there any other way to explain it mathematically to a newbie ? Edit Can anybody explain me this excerpt from the ...
1
vote
1answer
62 views

Bagging models with different metaparameters versus cross validation?

If we have a model with a metaparameter C, the usual way to tune this parameter is via cross-validation (or building a validation set). The generalization error of the model with the optimal is ...
2
votes
1answer
104 views

Overfit in aggregated models: boosting versus simple bagging

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the ...
2
votes
1answer
136 views

What is the purpose of using duplicated data in resampling techniques (e.g., bagging/bootstrapping)?

With bootstrapping and bagging, we resample from the dataset and end up building a model or estimating some sample statistic using the sampled data, which typically consists of at least $33\%$ ...
1
vote
1answer
20 views

Some ambiguities about used training dataset in bagged trees

I have some ambiguity about dividing training dataset in Bagging tree. In fact I have found in this article About Decision Tree Ensembles- Bagging That : the idea is to create several subsets of data ...
0
votes
2answers
48 views

Is boosting and bagging only relevant in the context of decision trees?

In the documents I've seen on boosting and bagging, it seems that they're always talked about in the context of decision trees. What are some other methods in which the two are applicable?
0
votes
0answers
26 views

Are predictions generated from bootstrap resampling normally distributed?

If I resample a dataset hundreds of times with replacement (bootstrap replications), fit some Model A on each bootstrap sample, and generate predictions (time series) from each fitted model, can I ...
0
votes
0answers
45 views

How are bagging and boosting methods of feature extraction/selection?

I am learning about boosting and bagging from the Amazon Web Services Machine Learning courses. In it, they describe bagging and boosting as ways to automate feature extraction and selection. My ...
3
votes
2answers
105 views

Bagging - Size of the aggregate bags?

I'm reading up on bagging (boostrap aggregation), and several sources seem to state that the size of the bags (consist of random sampling from our training set with replacement) is typically around 63%...
2
votes
1answer
17 views

Bagging sample probability

I know that the probability that an observation does not appear in the sample is $\left(1 - \frac{\ 1}{n} \right)^n ≈ \left(\frac{1}{e}\right) $ (when n is large). My question is how to find the ...
0
votes
0answers
280 views

Time complexity of bagging and random forest

Most sources and books state that time complexity of a single decision tree for n points and d dimensions (features) is $O(d * n^2 * log(n))$, with clever caching and one time sorting it’s $O(d * n * ...
2
votes
1answer
104 views

Why are boosted trees difficult to interpret?

I might not fully understand the topic but: Bootstrapped/bagged trees are difficult to interpret because the decisions are made from averaging the prediction of possibly hundreds of trees (ensemble). ...
0
votes
0answers
7 views

Artificial “Bagging” for GLM poisson

Let say I'm building a model to predict the number of accidents in a insurance portfolio of automobiles. The problem is that my model is is very very sensitive to the seed (for each time I change the ...
2
votes
0answers
9 views

Random forest and bagging: if both generate deep, non-pruned trees then what's the difference?

I understand that random forest only considers a subset of predictors at each node, but if you are generating deep trees then you'll eventually include all predictors in your model, the only ...
1
vote
1answer
90 views

Decision tree-based Bagging and Random Forest

I am new to Machine Learning and would like to know is it always true that decision tree-based bagging has worse predictive performance and is slower running time than random forest?
1
vote
0answers
26 views

Why do correlated decision trees increase bias?

To decrease variance in decision trees often, bagging is used to create several decision trees. This has better accuracy than one tree since the decrease in variance outweighs the increase in bias. ...
2
votes
1answer
74 views

Why is the error rate from bagging trees much higher than that from a single tree?

I'm running the classification method Bagging Tree (Bootstrap Aggregation) and compare the misclassification error rate with one from one single tree. We expect that the result from bagging tree is ...
0
votes
0answers
28 views

What is the meaning of component err.rate of class randomForest?

I asked this question on Stackoverflow, but it's likely that I will received no answer on that site. So I cross-post my question here. I'm using the function ...
4
votes
1answer
949 views

Why do we use random sample with replacement while implementing random forest?

Let's say we want to build random forest. Wikipedia says that we use random sample with replacement to do bagging. I don't understand why we can't use random sample without replacement.
0
votes
1answer
121 views

Bagging results from multiple k-fold cross-validation runs to estimate performance

Given a binary classification problem and a model construction algorithm, suppose I run, as an example, 5 fold (stratified) cross-validation 4 times. I can think of these as 5x4 = 20 training and 20 ...
1
vote
0answers
26 views

Trying to improve the generalization ability of decision tree with bagging or random forest

We are fitting a regression tree on a sample of about 5000 observations with 5 predictors. The data stems from 2 sources. There are reasons to suspect that the tree will not have exaggerated ...
0
votes
1answer
49 views

Machine Learning - Bagging

I am very new to Machine Learning I know i should use Decision Tree for bagging model because of its high variance. why cant we use any other algorithms for bagging ? if we can use any other ...
3
votes
1answer
461 views

Why is bagging stable classifiers not a good idea?

Citing Bagging Predictors, Section 4 (emphasis mine): Let each $(y, \mathbf{x})$ case in $\mathcal{L}$ be independently drawn from the distribution $P$. Suppose $y$ is numerical and $\phi(\mathbf{x}...
1
vote
1answer
335 views

why does random forest trees need to be deeper than gradient boosting trees

in Elements of Statistical Learning chapter 15. Random Forest, we see authors' note on RF v.s. GBT. One of them is that at 1000 terms, GBM depth 4 has smaller error than RF depth 6. Also we notice RF ...
0
votes
2answers
75 views

Using bagging and random forests together

I was looking at a kernel implementation (for text classification) and the following piece of code got me a little bit confused (I removed part of the features - in order to keep it light - as most of ...
2
votes
1answer
680 views
1
vote
0answers
17 views

Bagging Classifiers

In Bootstrap aggregating, instances that have not been drawn (by resampling with replacement), e.g. end up for example in the test set. However, in the bootstrap sample not every instance is unique. ...
1
vote
0answers
52 views

Does it make sense to use bagging with least square regression? if so, why?

We know that bagging can reduce the variance of the estimator, while keep the bias around the same. If we use bagging to ensemble multiple least square regressors, then are we going to reduce the ...
3
votes
1answer
989 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
7
votes
1answer
620 views

Variance of average of $n$ correlated random variables

Reading about deep leaning, I came across the following formula. $$ \mbox{var} \left( \frac{1}{n} \sum_{i=1}^{n} X_i \right) = \rho \sigma^2 + \frac{1-\rho}{n} \sigma^2 $$ where $X_1, \dots, X_n$ ...
0
votes
1answer
57 views

Is Gini index actively involved in splitting a Random Forest node?

Since there are many references that a RF uses a slightly different approach on splitting a node in comparison to Vanilla Bagging. Does Gini index play an active role in the split or it's just another ...
0
votes
1answer
139 views

Is bagging involved in the split of node of a tree of a Random Forest?

I know that using Bagging method in a RF, implies that the subset we give to the root node of each tree, has randomly selected Features and Attributes. I also know that during the split of a node ...
-1
votes
1answer
198 views

Parallel Bagging in supervised learning

How can we parallelize Bootstrap aggregation, a.k.a Bagging, i.e. train all classifiers at once?
9
votes
1answer
4k views

How can we explain the fact that “Bagging reduces the variance while retaining the bias” mathematically?

I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias". What is the mathematically principle behind this intution? I checked with few experts ...
14
votes
7answers
3k views

Random Forest and Decision Tree Algorithm

A random forest is a collection of decision trees following the bagging concept. When we move from one decision tree to the next decision tree then how does the information learned by last decision ...
1
vote
1answer
127 views

Aggregating method used by BaggingClassifier and BaggingRegressor?

What aggregating method is used by BaggingClassifier and BaggingRegressor in sklearn? Do they use soft voting or hard voting or averaging? The sklearn docs don't seem to clearly specify it.
6
votes
1answer
1k views

Why does bagging increase bias?

In machine learning, why does bagging increase bias? I've read that using less data would lead to a worse estimate of the parameters, but isn't the expected value of the parameter constant regardless ...