Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Zero inflated dependent variable and tree ml regression models

I wonder, if tree based ml models (e.g. xgb or random forests) are actually susceptible to zero inflated dependent variables (DVs) in the case of regression (in a sense the DV is at least bi-modal)? ...
  • 1,696
0 votes
0 answers
14 views

Bagging with correlated prediction models

Bagging works best for Ensembles that have a low bias and high variance, such as deep and complex decision trees, when the errors of the models in the ensemble are uncorrelated. I have a formal proof ...
  • 321
0 votes
1 answer
34 views

Subset Differences between Bagging, Random Forest, Boosting?

Per my understanding, there are 2 kinds of "subsets" that can be used when creating trees: 1) Subset of the dataset, 2) Subset of the features used per split. The concepts that I'm comparing ...
  • 422
1 vote
1 answer
114 views

Which of the following is FALSE about bagging?

I read this question somewhere(one statement is apparently false) and all four seem to be TRUE to me. What am I missing? What statement is FALSE? (Sharing my thoughts below the statements) Bagging is ...
  • 11
1 vote
0 answers
12 views

Can someone explain why finding similar embeddings coming from two different net gives bad recall?

I'm currently working on an ensemble of 5 differently trained networks using MinkLoc3D v2 as base-net. I'm currently investigating the reason for lousy recall when I compare the extracted database ...
0 votes
0 answers
25 views

How can a bagged model have lower bias than the individual predictors?

In class, it was mentioned that the individual predictors can have higher bias than the bagged model. To me, this is unintuitive as a key motivation for bagging is that the original model has ...
  • 21
0 votes
0 answers
17 views

How is bootstrapping used in machine learning? [duplicate]

I understood bootstrapping in the statistical context. Example: we have a sample of 1000 people. We want to know their mean. We pick 5 people at random (with replacement) for 20 times and we compute ...
2 votes
1 answer
181 views

Average Error vs. Aggregate Error

I was reading this paper on the history of Bagging Estimators (https://www.stat.berkeley.edu/~breiman/bagging.pdf) and came across the following section: I am having difficulty understanding the ...
  • 6,872
1 vote
0 answers
16 views

Question about an equation on bagging in Elements of Statistical Learning book [closed]

This should be a simple question but I must have missed something. Equation (8.52) of Section 8.7 Bagging on page 285 of Trevor Hastie, The Elements of Statistical Learning: Data Mining, Inference, ...
  • 915
0 votes
0 answers
208 views

Bagging with SVM and Neural Networks in R with caret

I am fairly new to the bagging technique and Caret's bagControl() as well as bag() and am currently trying to build an ensemble ...
  • 1
0 votes
0 answers
36 views

Bias, Variance and Bagging and what it means in relation to representational complexity

Re-looking at some basic ML algorithms bagging comes up along with the Bias-Variance trade off. I am confused on how bagging relates to representational capacity. Based upon the arguments I have read, ...
0 votes
0 answers
17 views

Regression Trees

Which is better over the other two? Random Forest, Bagging, or Boosting the tree-based method? My understanding is, that even though all three have their own preferred requirements to perform better. ...
  • 1
2 votes
1 answer
104 views

Does bagging work for OLS to improve prediction?

In Elements of Statistical Learning, section 8.7, the author states that The bagged estimate will differ from original estimate only when the latter is a nonlinear or adaptive function of the data I ...
  • 311
0 votes
0 answers
141 views

Why does Random Forest perform worse than Bagging?

The 2 results I got for bagging and random forest are shown below. It seems that calculating mean MSE from bootstrapping also result in a lower mean MSE for bagging as compared to random forest. Is ...
0 votes
3 answers
358 views

Why does test MSE always decrease with increasing training size (and decreasing test size)?

Context: I am trying to find the best predictive model for a dataset with 1000 observations. The problem is I am not sure what the best training and test size should be. So what I did was that I ran ...
1 vote
1 answer
189 views

Random Forest - varying seed to quantify uncertainty

I would like to quantify the uncertainty of a Random Forest binary classifier. The idea that popped in my mind was to fit the Random Forest 100 times with different seeds. Computing the variance of ...
0 votes
0 answers
14 views

Can I use the learning set as a test set for bootstrapped predictors?

This question relates to Leo Breiman's paper: Bagging Predictors from 1996. The author claims that if bagging is deployed, the original training set can be used to assess the performance of the ...
1 vote
1 answer
24 views

How does the performance of bagging depend on instability?

This question relates to Leo Breiman's paper: Bagging Predictors from 1996. Assuming that $\mathcal{L}$ denotes the training set and $\phi$ the predictor which depends on the training set and the ...
1 vote
0 answers
27 views

Lowering OLS prediction error with bootstrapping/bagging - a free lunch?

From what I understand, bagging reduces variance of prediction for a model. Though OLS is on the "low variance" part of the spectrum, I wish to understand anyway the implications of ...
  • 129
1 vote
0 answers
46 views

Bagging dependent data

Which are the possible caveats of using a Bagging algorithm (such as Random Forest), when data are not independent? Ensemble models usually exploit Bagging to reduce the variance by aggregating ...
1 vote
1 answer
47 views

proof out of bag evaluation [duplicate]

In Geron's book "Hands-on Machine Learning with Scikit-Learn and Tensorflow" there is this sentence on page 187 "By default a BaggingClassifier samples m training instances with ...
  • 155
0 votes
1 answer
434 views

How to bootstrap time series data

I have this dataset that contains multiple series (50 products). My dataset has 50 products (50 columns). each column contains the monthly sales of a product. I recently learned about bootstrap and ...
4 votes
1 answer
356 views

Boosting reduces bias when compared to what algorithm?

I am reading on bagging and boosting, and I understand how they both work (at least I think I do). I would like to talk in the context of decision tree ensembles as I think (not sure if correct) that ...
0 votes
0 answers
615 views

R Caret - Random Forest with repeated CV

I am currently struggling to explain the combination of bagging and cross-validation. I am aware of what each does separately but I have difficulties to explain how they are combined (if at all). For ...
  • 1
4 votes
1 answer
829 views

Is sampling with replacement better than sampling without replacement?

This question is more specific to machine learning. Is sampling with replacement good for random forests because it leaves some out of bag samples for testing or is it because it creates datasets/...
0 votes
0 answers
39 views

RandomForest with VST data

I used the randomForest() function in R for bagging without a training data set to identify the important features characterizing the members of groups A and B (binary testing). I get the following ...
6 votes
1 answer
99 views

Why is the size of the feature bagging sample typically the square root of the total predictor set size?

Applying feature sampling to each tree in the context of a Random Forest model, if the set of predictors is of size $p$, why does the size of the sample of predictors for each split typically be of ...
  • 61
0 votes
1 answer
208 views

Can we make bagging equivalent to random forest if we take max features in our own defined decision tree and pass it as estimator?

If in bagging we defined our own tuned decision tree as estimator with max features as parameter in that estimator rather than taking by default decision tree estimator available in BaggingClassifer ...
7 votes
2 answers
282 views

Intutition of why Bootstrap aggregating reduces overfitting?

Can somebody give me a non-mathematical intuition why Bootstrap aggregating reduces overfitting? From my point of view, we are not providing any additional information, we are not really enlengthen ...
  • 635
1 vote
1 answer
184 views

Why can't we sample without replacement for each tree in a random forest if the subsample size is large enough?

Usually if we have $n$ observations, for each tree with form a bootstrapped subsample of size $n$ with replacement. On googling it one common explanation I've seen is that with replacement sampling is ...
6 votes
2 answers
2k views

AdaBoost - why decision stumps instead of trees?

Since the original AdaBoost article it has been found out that boosting reduces both variance and bias in the classifier (in contrast to bagging, which only reduces variance). Original AdaBoost (and ...
  • 229
3 votes
2 answers
229 views

Elements of Statistical Learning: Variance Reduction via Bagging for Random Forests

While Reading chapter 15 of ESL about Random Forests I had some confusion with this phrase: The idea in random forests (Algorithm 15.1) is to improve the variance reduction of bagging by reducing the ...
0 votes
1 answer
80 views

Is there a model, apart from Random Forest, which uses bagging?

What are the popular machine learning algorithms which make use of bagging? Are there any apart from random forest?
  • 23
1 vote
0 answers
37 views

Regarding bagging, boosting and the NFL theorem

According to my understanding, bagging and boosting work in the following way: Bagging: Combine several high-variance/low-bias models to produce an ensemble model with lower variance and equal bias. ...
  • 23
0 votes
1 answer
34 views

Do tree methods like gradient boosting predict all iterations at once?

If I'm using a tree method (e.g GBM) and I have a time series hourly data, and I predict my target variable $y$ for the next 48 hours, do my predictions were made all at once, or does the second day ...
0 votes
1 answer
197 views

How does bagging effect linear model assumptions?

Linear regression has model assumptions. How does bagging effect model assumptions for linear regression? Also, should you build a bagged linear model with correlated and statistically significant ...
  • 1
2 votes
0 answers
246 views

Performing Bagging with the Lasso

I asked this question at StackOverflow but I think it might be better suited for here because I'm hoping to understand the general idea of bagging. I'm trying to implement bagging in ...
  • 233
1 vote
1 answer
85 views

bias-variance tradeoff is conflict to bagging

I first give the notations of bias-variance tradeoff: sample set: $D = \{(x_1,y_1),\cdots,(x_N,y_N)\}.$ relation between $y$ and $x:$ $y = f(x) + \epsilon.$ here $f(x)$ is a deterministic function; $...
  • 1,129
4 votes
2 answers
437 views

How many neglected samples when drawn with replacement? (bagging)

I learned a while ago about an interesting place that $e$ shows up in probability: if there are $n$ items and you sample $n$ times with replacement, you would expect that the fraction of samples that ...
  • 187
1 vote
0 answers
14 views

Is there a way to know which feature is causing a split or present in root or decision node of a decision tree in Random Forest using Python? [closed]

Let's say I have 100 features f1, f2, .... ,f100 and I want to know if f3, f4, f5 caused any split or were present in a root/decision node of any decision tree in a random forest. I have close to 200 ...
1 vote
1 answer
340 views

What is the limit of the random forest (or bagging) estimator?

I am looking for a proof or intuition as to why the absolute limit of a random forest estimator is the expectancy of a single tree (see citation below), i.e: $$ \hat{f}_{rf}(x) = \lim_{B \to \infty}[\...
  • 1,011
1 vote
0 answers
31 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
2 votes
1 answer
479 views

Is n_estimators in BaggingRegressor() the number of trees or data subsets?

I'm learning about using the BaggingRegressor() from scikit learn (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html) Its <...
  • 79
2 votes
1 answer
1k views

How does bagging reduce variance?

I read this answer. Was still unable to understand how bagging reduces variance. Is there any other way to explain it mathematically to a newbie ? Edit Can anybody explain me this excerpt from the ...
  • 119
1 vote
1 answer
90 views

Bagging models with different metaparameters versus cross validation?

If we have a model with a metaparameter C, the usual way to tune this parameter is via cross-validation (or building a validation set). The generalization error of the model with the optimal is ...
  • 748
2 votes
1 answer
302 views

Overfit in aggregated models: boosting versus simple bagging

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the ...
  • 748
5 votes
1 answer
773 views

What is the purpose of using duplicated data in resampling techniques (e.g., bagging/bootstrapping)?

With bootstrapping and bagging, we resample from the dataset and end up building a model or estimating some sample statistic using the sampled data, which typically consists of at least $33\%$ ...
  • 1,057
1 vote
1 answer
23 views

Some ambiguities about used training dataset in bagged trees

I have some ambiguity about dividing training dataset in Bagging tree. In fact I have found in this article About Decision Tree Ensembles- Bagging That : the idea is to create several subsets of data ...
  • 125
2 votes
2 answers
305 views

Is boosting and bagging only relevant in the context of decision trees?

In the documents I've seen on boosting and bagging, it seems that they're always talked about in the context of decision trees. What are some other methods in which the two are applicable?
  • 563
0 votes
0 answers
31 views

Are predictions generated from bootstrap resampling normally distributed?

If I resample a dataset hundreds of times with replacement (bootstrap replications), fit some Model A on each bootstrap sample, and generate predictions (time series) from each fitted model, can I ...
  • 3,305