Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

Filter by
Sorted by
Tagged with
2 votes
1 answer
28 views

Explanation for the success of bagging

I'm reading Machine Learning - A First Course for Engineers and Scientists. On page 168 they give a rough explanation for why bagging works. I'm a little confused by their explanation. They consider ...
oweydd's user avatar
  • 215
0 votes
0 answers
13 views

Ensemble, merge or combine multiple lmertree objects

Working with the PISA data, which includes multiple achievement scores (plausible values) for each participant, I would like to run the same lmertree and ensemble ...
Burak Aydin's user avatar
0 votes
1 answer
42 views

Bagging Ensemble Math

You are working on a binary classification problem with 3 input features and have chosen to apply a bagging algorithm (Algorithm X) on this data. You have set max_features = 2 and n_estimators = 3. ...
Tanjim Taharat Aurpa's user avatar
0 votes
0 answers
16 views

Help needed for interpretation of mtry and MSE calculation for bagging and random forests

I have a question regarding the mtry values for the two models Bagging and Random Forests. I applied the mtry measure for the California Housing Dataset and then for another dataset about white wine. ...
Marie_wue's user avatar
2 votes
1 answer
72 views

Are Bagged Ensembles of Neural Networks Actually Helpful?

I've been looking into ways to estimate uncertainty for regression tasks on neural networks. One of the obvious options is ensemble modeling. Consider an ensemble of neural networks that all have ...
noNameTed's user avatar
  • 113
0 votes
0 answers
79 views

OOB in Random Forests - Detailed Calculation

In the book An introduction to statistical learning, it is mentioned that: One can show that on average, each bagged tree makes use of around two-thirds of the observations. The remaining one-third ...
Kian's user avatar
  • 23
0 votes
0 answers
36 views

Using OOB Error Rate to improve Random Forest

I've been reading about the use of OOB Error Rate and its applications and use in Random Forest, for which some curiosity is how I could further use this value in the optimization of my own modelling ...
informaticianwannabe's user avatar
0 votes
0 answers
20 views

What is the correct calculation for AIC corrected (AICc) for a bagged random forest model using the Boston Housing data set? [duplicate]

This question of calculating AIC was answered for a specific linear model here: Calculating AIC “by hand” in R The problem and solution for a linear model are as follows: ...
Russ Conte's user avatar
2 votes
0 answers
39 views

Is bagging less useful in 'big data' settings?

In 'big data' settings where the number of samples $n$ may be very large (for fixed number of features), is bagging less or more effective at reducing variance? I heard the claim that it is less ...
WeakLearner's user avatar
  • 1,461
1 vote
0 answers
25 views

allocating specific training subset to specific classifier for ensemble mode

I have a dataset I'd like to break into 2 to 3 subsets to account for outliers. Train the subsets with individual classifier and combining them. Understand that Bagging uses subset of the base ...
ManOnTheMoon's user avatar
0 votes
0 answers
14 views

Why Out-of-bag score used for Bagging Ensembles and only for Bagging?

I study ensemble machine learning and I have noticed OOB (out-of-bag) score in some implementations. I understand the concept but I have some questions about the general usage: Why is it used for ...
Fredrik's user avatar
  • 725
0 votes
0 answers
66 views

Bootstrap option in sklearn.ensemble.RandomForestRegressor in Python

If we read the documentation about the sklearn.ensemble.RandomForestRegressor function, we find a bootstrap option: bootstrap : ...
TWest's user avatar
  • 347
1 vote
0 answers
67 views

Zero inflated dependent variable and tree ml regression models

I wonder, if tree based ml models (e.g. xgb or random forests) are actually susceptible to zero inflated dependent variables (DVs) in the case of regression (in a sense the DV is at least bi-modal)? ...
cs0815's user avatar
  • 2,115
0 votes
1 answer
225 views

Subset Differences between Bagging, Random Forest, Boosting?

Per my understanding, there are 2 kinds of "subsets" that can be used when creating trees: 1) Subset of the dataset, 2) Subset of the features used per split. The concepts that I'm comparing ...
Katsu's user avatar
  • 861
1 vote
1 answer
224 views

Which of the following is FALSE about bagging?

I read this question somewhere(one statement is apparently false) and all four seem to be TRUE to me. What am I missing? What statement is FALSE? (Sharing my thoughts below the statements) Bagging is ...
Akshit's user avatar
  • 11
1 vote
0 answers
15 views

Can someone explain why finding similar embeddings coming from two different net gives bad recall?

I'm currently working on an ensemble of 5 differently trained networks using MinkLoc3D v2 as base-net. I'm currently investigating the reason for lousy recall when I compare the extracted database ...
JackFrost67's user avatar
0 votes
0 answers
17 views

How is bootstrapping used in machine learning? [duplicate]

I understood bootstrapping in the statistical context. Example: we have a sample of 1000 people. We want to know their mean. We pick 5 people at random (with replacement) for 20 times and we compute ...
user1315621's user avatar
2 votes
1 answer
304 views

Average Error vs. Aggregate Error

I was reading this paper on the history of Bagging Estimators (https://www.stat.berkeley.edu/~breiman/bagging.pdf) and came across the following section: I am having difficulty understanding the ...
Uk rain troll's user avatar
1 vote
0 answers
20 views

Question about an equation on bagging in Elements of Statistical Learning book [closed]

This should be a simple question but I must have missed something. Equation (8.52) of Section 8.7 Bagging on page 285 of Trevor Hastie, The Elements of Statistical Learning: Data Mining, Inference, ...
Hans's user avatar
  • 995
2 votes
1 answer
246 views

Does bagging work for OLS to improve prediction?

In Elements of Statistical Learning, section 8.7, the author states that The bagged estimate will differ from original estimate only when the latter is a nonlinear or adaptive function of the data I ...
wwyws's user avatar
  • 321
0 votes
1 answer
257 views

Why does Random Forest perform worse than Bagging?

The 2 results I got for bagging and random forest are shown below. It seems that calculating mean MSE from bootstrapping also result in a lower mean MSE for bagging as compared to random forest. Is ...
stat_2021's user avatar
0 votes
3 answers
1k views

Why does test MSE always decrease with increasing training size (and decreasing test size)?

Context: I am trying to find the best predictive model for a dataset with 1000 observations. The problem is I am not sure what the best training and test size should be. So what I did was that I ran ...
stat_2021's user avatar
1 vote
1 answer
373 views

Random Forest - varying seed to quantify uncertainty

I would like to quantify the uncertainty of a Random Forest binary classifier. The idea that popped in my mind was to fit the Random Forest 100 times with different seeds. Computing the variance of ...
Niccolò Ajroldi's user avatar
1 vote
1 answer
26 views

How does the performance of bagging depend on instability?

This question relates to Leo Breiman's paper: Bagging Predictors from 1996. Assuming that $\mathcal{L}$ denotes the training set and $\phi$ the predictor which depends on the training set and the ...
crabnebul's user avatar
1 vote
0 answers
31 views

Lowering OLS prediction error with bootstrapping/bagging - a free lunch?

From what I understand, bagging reduces variance of prediction for a model. Though OLS is on the "low variance" part of the spectrum, I wish to understand anyway the implications of ...
Arshdeep's user avatar
  • 139
1 vote
0 answers
66 views

Bagging dependent data

Which are the possible caveats of using a Bagging algorithm (such as Random Forest), when data are not independent? Ensemble models usually exploit Bagging to reduce the variance by aggregating ...
Niccolò Ajroldi's user avatar
1 vote
1 answer
123 views

proof out of bag evaluation [duplicate]

In Geron's book "Hands-on Machine Learning with Scikit-Learn and Tensorflow" there is this sentence on page 187 "By default a BaggingClassifier samples m training instances with ...
NNN's user avatar
  • 155
0 votes
1 answer
760 views

How to bootstrap time series data

I have this dataset that contains multiple series (50 products). My dataset has 50 products (50 columns). each column contains the monthly sales of a product. I recently learned about bootstrap and ...
Abdel_DAS's user avatar
5 votes
1 answer
1k views

Boosting reduces bias when compared to what algorithm?

I am reading on bagging and boosting, and I understand how they both work (at least I think I do). I would like to talk in the context of decision tree ensembles as I think (not sure if correct) that ...
Slim Shady's user avatar
0 votes
0 answers
907 views

R Caret - Random Forest with repeated CV

I am currently struggling to explain the combination of bagging and cross-validation. I am aware of what each does separately but I have difficulties to explain how they are combined (if at all). For ...
Ru8zi9's user avatar
  • 1
4 votes
1 answer
1k views

Is sampling with replacement better than sampling without replacement?

This question is more specific to machine learning. Is sampling with replacement good for random forests because it leaves some out of bag samples for testing or is it because it creates datasets/...
Ixion Chowdhury's user avatar
0 votes
0 answers
45 views

RandomForest with VST data

I used the randomForest() function in R for bagging without a training data set to identify the important features characterizing the members of groups A and B (binary testing). I get the following ...
AennaPhD's user avatar
6 votes
1 answer
180 views

Why is the size of the feature bagging sample typically the square root of the total predictor set size?

Applying feature sampling to each tree in the context of a Random Forest model, if the set of predictors is of size $p$, why does the size of the sample of predictors for each split typically be of ...
tempMC's user avatar
  • 61
1 vote
1 answer
305 views

Can we make bagging equivalent to random forest if we take max features in our own defined decision tree and pass it as estimator?

If in bagging we defined our own tuned decision tree as estimator with max features as parameter in that estimator rather than taking by default decision tree estimator available in BaggingClassifer ...
Raufur Khan's user avatar
7 votes
2 answers
539 views

Intutition of why Bootstrap aggregating reduces overfitting?

Can somebody give me a non-mathematical intuition why Bootstrap aggregating reduces overfitting? From my point of view, we are not providing any additional information, we are not really enlengthen ...
J3lackkyy's user avatar
  • 655
3 votes
2 answers
552 views

R Squared (OOB) and R Square from correlation of prediction of test set is different?

I'm using simulated data and fitting Random Forest model for regression on a training dataset. What is confusing me is that after running Random Forest, I got R Squared (OOB) equal to 0.14. But when I ...
Atiq Ur Rehman's user avatar
1 vote
1 answer
235 views

Why can't we sample without replacement for each tree in a random forest if the subsample size is large enough?

Usually if we have $n$ observations, for each tree with form a bootstrapped subsample of size $n$ with replacement. On googling it one common explanation I've seen is that with replacement sampling is ...
user9343456's user avatar
6 votes
2 answers
3k views

AdaBoost - why decision stumps instead of trees?

Since the original AdaBoost article it has been found out that boosting reduces both variance and bias in the classifier (in contrast to bagging, which only reduces variance). Original AdaBoost (and ...
qalis's user avatar
  • 239
3 votes
2 answers
367 views

Elements of Statistical Learning: Variance Reduction via Bagging for Random Forests

While Reading chapter 15 of ESL about Random Forests I had some confusion with this phrase: The idea in random forests (Algorithm 15.1) is to improve the variance reduction of bagging by reducing the ...
Christian Singer's user avatar
0 votes
1 answer
118 views

Is there a model, apart from Random Forest, which uses bagging?

What are the popular machine learning algorithms which make use of bagging? Are there any apart from random forest?
nyptop's user avatar
  • 23
1 vote
0 answers
41 views

Regarding bagging, boosting and the NFL theorem

According to my understanding, bagging and boosting work in the following way: Bagging: Combine several high-variance/low-bias models to produce an ensemble model with lower variance and equal bias. ...
Enk9456's user avatar
  • 33
0 votes
1 answer
36 views

Do tree methods like gradient boosting predict all iterations at once?

If I'm using a tree method (e.g GBM) and I have a time series hourly data, and I predict my target variable $y$ for the next 48 hours, do my predictions were made all at once, or does the second day ...
Numbermind's user avatar
2 votes
1 answer
517 views

How does bagging affect linear model assumptions?

Linear regression has assumptions. How does bagging affect model assumptions for linear regression? Also, should you build a bagged linear model with correlated and statistically significant variables ...
reeves's user avatar
  • 21
2 votes
0 answers
414 views

Performing Bagging with the Lasso

I asked this question at StackOverflow but I think it might be better suited for here because I'm hoping to understand the general idea of bagging. I'm trying to implement bagging in ...
AW27's user avatar
  • 233
1 vote
1 answer
156 views

bias-variance tradeoff is conflict to bagging

I first give the notations of bias-variance tradeoff: sample set: $D = \{(x_1,y_1),\cdots,(x_N,y_N)\}.$ relation between $y$ and $x:$ $y = f(x) + \epsilon.$ here $f(x)$ is a deterministic function; $...
user6703592's user avatar
  • 1,305
4 votes
2 answers
730 views

How many neglected samples when drawn with replacement? (bagging)

I learned a while ago about an interesting place that $e$ shows up in probability: if there are $n$ items and you sample $n$ times with replacement, you would expect that the fraction of samples that ...
efthimio's user avatar
  • 197
1 vote
0 answers
14 views

Is there a way to know which feature is causing a split or present in root or decision node of a decision tree in Random Forest using Python? [closed]

Let's say I have 100 features f1, f2, .... ,f100 and I want to know if f3, f4, f5 caused any split or were present in a root/decision node of any decision tree in a random forest. I have close to 200 ...
user_beginner's user avatar
1 vote
1 answer
459 views

What is the limit of the random forest (or bagging) estimator?

I am looking for a proof or intuition as to why the absolute limit of a random forest estimator is the expectancy of a single tree (see citation below), i.e: $$ \hat{f}_{rf}(x) = \lim_{B \to \infty}[\...
PaulG's user avatar
  • 1,245
1 vote
0 answers
43 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
DiamondDust's user avatar
2 votes
1 answer
839 views

Is n_estimators in BaggingRegressor() the number of trees or data subsets?

I'm learning about using the BaggingRegressor() from scikit learn (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html) Its <...
DN1's user avatar
  • 111