Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

Filter by
Sorted by
Tagged with
1
vote
0answers
10 views

Is there a way to know which feature is causing a split or present in root or decision node of a decision tree in Random Forest using Python? [closed]

Let's say I have 100 features f1, f2, .... ,f100 and I want to know if f3, f4, f5 caused any split or were present in a root/decision node of any decision tree in a random forest. I have close to 200 ...
0
votes
0answers
18 views

Stacking of Machine Learning models using different data splits

I have one single dataset with 2 classes. I want to make a model for binary classification, and I am experimenting a bit. My intention is to use stacking on some models by using subsets of the 1 ...
1
vote
1answer
48 views

What is the limit of the random forest (or bagging) estimator?

I am looking for a proof or intuition as to why the absolute limit of a random forest estimator is the expectancy of a single tree (see citation below), i.e: $$ \hat{f}_{rf}(x) = \lim_{B \to \infty}[\...
1
vote
0answers
16 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
2
votes
1answer
22 views

Is n_estimators in BaggingRegressor() the number of trees or data subsets?

I'm learning about using the BaggingRegressor() from scikit learn (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html) Its <...
2
votes
1answer
117 views

How does bagging reduce variance?

I read this answer. Was still unable to understand how bagging reduces variance. Is there any other way to explain it mathematically to a newbie ? Edit Can anybody explain me this excerpt from the ...
0
votes
1answer
53 views

Bagging models with different metaparameters versus cross validation?

If we have a model with a metaparameter C, the usual way to tune this parameter is via cross-validation (or building a validation set). The generalization error of the model with the optimal is ...
2
votes
1answer
70 views

Overfit in aggregated models: boosting versus simple bagging

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the ...
1
vote
1answer
66 views

What is the purpose of using duplicated data in resampling techniques (e.g., bagging/bootstrapping)?

With bootstrapping and bagging, we resample from the dataset and end up building a model or estimating some sample statistic using the sampled data, which typically consists of at least $33\%$ ...
1
vote
1answer
20 views

Some ambiguities about used training dataset in bagged trees

I have some ambiguity about dividing training dataset in Bagging tree. In fact I have found in this article About Decision Tree Ensembles- Bagging That : the idea is to create several subsets of data ...
0
votes
2answers
43 views

Is boosting and bagging only relevant in the context of decision trees?

In the documents I've seen on boosting and bagging, it seems that they're always talked about in the context of decision trees. What are some other methods in which the two are applicable?
0
votes
0answers
26 views

Are predictions generated from bootstrap resampling normally distributed?

If I resample a dataset hundreds of times with replacement (bootstrap replications), fit some Model A on each bootstrap sample, and generate predictions (time series) from each fitted model, can I ...
0
votes
0answers
32 views

How are bagging and boosting methods of feature extraction/selection?

I am learning about boosting and bagging from the Amazon Web Services Machine Learning courses. In it, they describe bagging and boosting as ways to automate feature extraction and selection. My ...
2
votes
2answers
56 views

Bagging - Size of the aggregate bags?

I'm reading up on bagging (boostrap aggregation), and several sources seem to state that the size of the bags (consist of random sampling from our training set with replacement) is typically around 63%...
2
votes
1answer
15 views

Bagging sample probability

I know that the probability that an observation does not appear in the sample is $\left(1 - \frac{\ 1}{n} \right)^n ≈ \left(\frac{1}{e}\right) $ (when n is large). My question is how to find the ...
0
votes
0answers
147 views

Time complexity of bagging and random forest

Most sources and books state that time complexity of a single decision tree for n points and d dimensions (features) is $O(d * n^2 * log(n))$, with clever caching and one time sorting it’s $O(d * n * ...
2
votes
1answer
99 views

Why are boosted trees difficult to interpret?

I might not fully understand the topic but: Bootstrapped/bagged trees are difficult to interpret because the decisions are made from averaging the prediction of possibly hundreds of trees (ensemble). ...
0
votes
0answers
6 views

Artificial “Bagging” for GLM poisson

Let say I'm building a model to predict the number of accidents in a insurance portfolio of automobiles. The problem is that my model is is very very sensitive to the seed (for each time I change the ...
2
votes
0answers
9 views

Random forest and bagging: if both generate deep, non-pruned trees then what's the difference?

I understand that random forest only considers a subset of predictors at each node, but if you are generating deep trees then you'll eventually include all predictors in your model, the only ...
0
votes
0answers
13 views

Evaluating baggedETS on a Test set

I'm using the forecast package to test a variety of models on monthly sales for 400 products sold by my company. I'm following the practice of fitting on a test ...
1
vote
1answer
45 views

Decision tree-based Bagging and Random Forest

I am new to Machine Learning and would like to know is it always true that decision tree-based bagging has worse predictive performance and is slower running time than random forest?
1
vote
0answers
25 views

Why do correlated decision trees increase bias?

To decrease variance in decision trees often, bagging is used to create several decision trees. This has better accuracy than one tree since the decrease in variance outweighs the increase in bias. ...
2
votes
1answer
53 views

Why is the error rate from bagging trees much higher than that from a single tree?

I'm running the classification method Bagging Tree (Bootstrap Aggregation) and compare the misclassification error rate with one from one single tree. We expect that the result from bagging tree is ...
0
votes
0answers
24 views

What is the meaning of component err.rate of class randomForest?

I asked this question on Stackoverflow, but it's likely that I will received no answer on that site. So I cross-post my question here. I'm using the function ...
0
votes
0answers
19 views

Why is out of bag error higher than out of fold error?

There are two reasosns that i have come across for out of bag error being higher than the expected OOF error. these are: Out of bag predictions are on a subset of the forest for which a particular ...
2
votes
1answer
572 views

Why do we use random sample with replacement while implementing random forest?

Let's say we want to build random forest. Wikipedia says that we use random sample with replacement to do bagging. I don't understand why we can't use random sample without replacement.
0
votes
1answer
101 views

Bagging results from multiple k-fold cross-validation runs to estimate performance

Given a binary classification problem and a model construction algorithm, suppose I run, as an example, 5 fold (stratified) cross-validation 4 times. I can think of these as 5x4 = 20 training and 20 ...
1
vote
0answers
25 views

Trying to improve the generalization ability of decision tree with bagging or random forest

We are fitting a regression tree on a sample of about 5000 observations with 5 predictors. The data stems from 2 sources. There are reasons to suspect that the tree will not have exaggerated ...
0
votes
1answer
47 views

Machine Learning - Bagging

I am very new to Machine Learning I know i should use Decision Tree for bagging model because of its high variance. why cant we use any other algorithms for bagging ? if we can use any other ...
3
votes
1answer
393 views

Why is bagging stable classifiers not a good idea?

Citing Bagging Predictors, Section 4 (emphasis mine): Let each $(y, \mathbf{x})$ case in $\mathcal{L}$ be independently drawn from the distribution $P$. Suppose $y$ is numerical and $\phi(\mathbf{x}...
1
vote
1answer
244 views

why does random forest trees need to be deeper than gradient boosting trees

in Elements of Statistical Learning chapter 15. Random Forest, we see authors' note on RF v.s. GBT. One of them is that at 1000 terms, GBM depth 4 has smaller error than RF depth 6. Also we notice RF ...
0
votes
2answers
73 views

Using bagging and random forests together

I was looking at a kernel implementation (for text classification) and the following piece of code got me a little bit confused (I removed part of the features - in order to keep it light - as most of ...
2
votes
1answer
508 views
1
vote
0answers
17 views

Bagging Classifiers

In Bootstrap aggregating, instances that have not been drawn (by resampling with replacement), e.g. end up for example in the test set. However, in the bootstrap sample not every instance is unique. ...
1
vote
0answers
47 views

Does it make sense to use bagging with least square regression? if so, why?

We know that bagging can reduce the variance of the estimator, while keep the bias around the same. If we use bagging to ensemble multiple least square regressors, then are we going to reduce the ...
3
votes
1answer
745 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
6
votes
1answer
389 views

Variance of average of $n$ correlated random variables

Reading about deep leaning, I came across the following formula. $$ \mbox{var} \left( \frac{1}{n} \sum_{i=1}^{n} X_i \right) = \rho \sigma^2 + \frac{1-\rho}{n} \sigma^2 $$ where $X_1, \dots, X_n$ ...
0
votes
1answer
56 views

Is Gini index actively involved in splitting a Random Forest node?

Since there are many references that a RF uses a slightly different approach on splitting a node in comparison to Vanilla Bagging. Does Gini index play an active role in the split or it's just another ...
0
votes
1answer
130 views

Is bagging involved in the split of node of a tree of a Random Forest?

I know that using Bagging method in a RF, implies that the subset we give to the root node of each tree, has randomly selected Features and Attributes. I also know that during the split of a node ...
-1
votes
1answer
180 views

Parallel Bagging in supervised learning

How can we parallelize Bootstrap aggregation, a.k.a Bagging, i.e. train all classifiers at once?
9
votes
1answer
4k views

How can we explain the fact that “Bagging reduces the variance while retaining the bias” mathematically?

I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias". What is the mathematically principle behind this intution? I checked with few experts ...
14
votes
7answers
3k views

Random Forest and Decision Tree Algorithm

A random forest is a collection of decision trees following the bagging concept. When we move from one decision tree to the next decision tree then how does the information learned by last decision ...
1
vote
1answer
127 views

Aggregating method used by BaggingClassifier and BaggingRegressor?

What aggregating method is used by BaggingClassifier and BaggingRegressor in sklearn? Do they use soft voting or hard voting or averaging? The sklearn docs don't seem to clearly specify it.
6
votes
1answer
1k views

Why does bagging increase bias?

In machine learning, why does bagging increase bias? I've read that using less data would lead to a worse estimate of the parameters, but isn't the expected value of the parameter constant regardless ...
18
votes
1answer
3k views

Boosting AND Bagging Trees (XGBoost, LightGBM)

There are many blog posts, YouTube videos, etc. about the ideas of bagging or boosting trees. My general understanding is that the pseudo code for each is: Bagging: Take N random samples of x% of ...
2
votes
1answer
443 views

Random Forest: Number of Bags vs. Number of Trees

What are the differences between the number of bags and the number of trees in a Random Forest? For what I knew number of bags = number of tress in RF. However, in certain packages such as RankLib, ...
1
vote
0answers
426 views

What is the relation between minimum instances per node and max depth?

In bagging and boosting models like random forest and xgboost we have hyper-parameters like minimum instances per node and max depth. If max depth is high the minimum instances per node will be less ...
3
votes
2answers
2k views

What are advantages of random forests vs using bagging with other classifiers?

I'm studying Random Forests, but after reviewing the methods I got the following line of reasoning: I feel like the big advantage of random forests happens in the bagging process where nearly ...
0
votes
1answer
30 views

Why does creating training sets with replacement lead to better performance?

Pasting generally suffers from lower performance than bagging, because it training sets without replacement. Why does creating training sets with replacement lead to better performance?
1
vote
1answer
44 views

Sample size for designing a study which creates a classifier

I have to plan a study in which I will have to create a classifier. The output variable is a binary with an estimated proportion of value 1 in the overall population of interest to be 0.10 (and ...