Skip to main content

Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

Model parameter averaging in Bagging

I wonder if the following bagging method is used in practice, or at least any reference for this. Assume that we sample (sub)-datasets from the original training set, and train $n$ many logistic ...
Seewoo Lee's user avatar
0 votes
0 answers
32 views

Evaluation of BaggingClassifier using Out-of-bag observations is an indirect evaluation of bagging classifier?

May I know is the following correct? When evaluating a BaggingClassifier using out-of-bag (OOB) samples, the goal is indeed to assess the performance of the ensemble as a whole, rather than focusing ...
jiang xi's user avatar
0 votes
0 answers
9 views

What are the problem with running binary classification on panel data using logistic regression?

I am a newbie and I am trying to understand why one cannot run algorithms like logistic regression or decision trees on panel dataset ? I am running an employee attrition model where the dependent ...
learner's user avatar
  • 617
3 votes
1 answer
34 views

Explanation for the success of bagging

I'm reading Machine Learning - A First Course for Engineers and Scientists. On page 168 they give a rough explanation for why bagging works. I'm a little confused by their explanation. They consider ...
oweydd's user avatar
  • 235
0 votes
0 answers
13 views

Ensemble, merge or combine multiple lmertree objects

Working with the PISA data, which includes multiple achievement scores (plausible values) for each participant, I would like to run the same lmertree and ensemble ...
Burak Aydin's user avatar
0 votes
1 answer
60 views

Bagging Ensemble Math

You are working on a binary classification problem with 3 input features and have chosen to apply a bagging algorithm (Algorithm X) on this data. You have set max_features = 2 and n_estimators = 3. ...
Tanjim Taharat Aurpa's user avatar
0 votes
0 answers
17 views

Help needed for interpretation of mtry and MSE calculation for bagging and random forests

I have a question regarding the mtry values for the two models Bagging and Random Forests. I applied the mtry measure for the California Housing Dataset and then for another dataset about white wine. ...
Marie_wue's user avatar
2 votes
1 answer
193 views

Are Bagged Ensembles of Neural Networks Actually Helpful?

I've been looking into ways to estimate uncertainty for regression tasks on neural networks. One of the obvious options is ensemble modeling. Consider an ensemble of neural networks that all have ...
noNameTed's user avatar
  • 135
0 votes
0 answers
114 views

OOB in Random Forests - Detailed Calculation

In the book An introduction to statistical learning, it is mentioned that: One can show that on average, each bagged tree makes use of around two-thirds of the observations. The remaining one-third ...
Kian's user avatar
  • 23
0 votes
0 answers
38 views

Using OOB Error Rate to improve Random Forest

I've been reading about the use of OOB Error Rate and its applications and use in Random Forest, for which some curiosity is how I could further use this value in the optimization of my own modelling ...
informaticianwannabe's user avatar
2 votes
0 answers
49 views

Is bagging less useful in 'big data' settings?

In 'big data' settings where the number of samples $n$ may be very large (for fixed number of features), is bagging less or more effective at reducing variance? I heard the claim that it is less ...
WeakLearner's user avatar
  • 1,501
1 vote
0 answers
28 views

allocating specific training subset to specific classifier for ensemble mode

I have a dataset I'd like to break into 2 to 3 subsets to account for outliers. Train the subsets with individual classifier and combining them. Understand that Bagging uses subset of the base ...
ManOnTheMoon's user avatar
3 votes
0 answers
106 views

Zero inflated dependent variable and tree ml regression models

I wonder, if tree based ml models (e.g. xgb or random forests) are actually susceptible to zero inflated dependent variables (DVs) in the case of regression (in a sense the DV is at least bi-modal)? ...
cs0815's user avatar
  • 2,245
1 vote
1 answer
320 views

Subset Differences between Bagging, Random Forest, Boosting?

Per my understanding, there are 2 kinds of "subsets" that can be used when creating trees: 1) Subset of the dataset, 2) Subset of the features used per split. The concepts that I'm comparing ...
Katsu's user avatar
  • 1,011
1 vote
1 answer
245 views

Which of the following is FALSE about bagging?

I read this question somewhere(one statement is apparently false) and all four seem to be TRUE to me. What am I missing? What statement is FALSE? (Sharing my thoughts below the statements) Bagging is ...
Akshit's user avatar
  • 11
1 vote
0 answers
15 views

Can someone explain why finding similar embeddings coming from two different net gives bad recall?

I'm currently working on an ensemble of 5 differently trained networks using MinkLoc3D v2 as base-net. I'm currently investigating the reason for lousy recall when I compare the extracted database ...
JackFrost67's user avatar
0 votes
0 answers
18 views

How is bootstrapping used in machine learning? [duplicate]

I understood bootstrapping in the statistical context. Example: we have a sample of 1000 people. We want to know their mean. We pick 5 people at random (with replacement) for 20 times and we compute ...
user1315621's user avatar
2 votes
1 answer
354 views

Average Error vs. Aggregate Error

I was reading this paper on the history of Bagging Estimators (https://www.stat.berkeley.edu/~breiman/bagging.pdf) and came across the following section: I am having difficulty understanding the ...
stats_noob's user avatar
1 vote
0 answers
20 views

Question about an equation on bagging in Elements of Statistical Learning book [closed]

This should be a simple question but I must have missed something. Equation (8.52) of Section 8.7 Bagging on page 285 of Trevor Hastie, The Elements of Statistical Learning: Data Mining, Inference, ...
Hans's user avatar
  • 1,015
2 votes
1 answer
285 views

Does bagging work for OLS to improve prediction?

In Elements of Statistical Learning, section 8.7, the author states that The bagged estimate will differ from original estimate only when the latter is a nonlinear or adaptive function of the data I ...
wwyws's user avatar
  • 321
0 votes
1 answer
276 views

Why does Random Forest perform worse than Bagging?

The 2 results I got for bagging and random forest are shown below. It seems that calculating mean MSE from bootstrapping also result in a lower mean MSE for bagging as compared to random forest. Is ...
stat_2021's user avatar
0 votes
3 answers
2k views

Why does test MSE always decrease with increasing training size (and decreasing test size)?

Context: I am trying to find the best predictive model for a dataset with 1000 observations. The problem is I am not sure what the best training and test size should be. So what I did was that I ran ...
stat_2021's user avatar
1 vote
1 answer
405 views

Random Forest - varying seed to quantify uncertainty

I would like to quantify the uncertainty of a Random Forest binary classifier. The idea that popped in my mind was to fit the Random Forest 100 times with different seeds. Computing the variance of ...
Niccolò Ajroldi's user avatar
1 vote
1 answer
27 views

How does the performance of bagging depend on instability?

This question relates to Leo Breiman's paper: Bagging Predictors from 1996. Assuming that $\mathcal{L}$ denotes the training set and $\phi$ the predictor which depends on the training set and the ...
crabnebul's user avatar
1 vote
0 answers
34 views

Lowering OLS prediction error with bootstrapping/bagging - a free lunch?

From what I understand, bagging reduces variance of prediction for a model. Though OLS is on the "low variance" part of the spectrum, I wish to understand anyway the implications of ...
Arshdeep's user avatar
  • 139
1 vote
0 answers
71 views

Bagging dependent data

Which are the possible caveats of using a Bagging algorithm (such as Random Forest), when data are not independent? Ensemble models usually exploit Bagging to reduce the variance by aggregating ...
Niccolò Ajroldi's user avatar
1 vote
1 answer
145 views

proof out of bag evaluation [duplicate]

In Geron's book "Hands-on Machine Learning with Scikit-Learn and Tensorflow" there is this sentence on page 187 "By default a BaggingClassifier samples m training instances with ...
NNN's user avatar
  • 155
0 votes
1 answer
823 views

How to bootstrap time series data

I have this dataset that contains multiple series (50 products). My dataset has 50 products (50 columns). each column contains the monthly sales of a product. I recently learned about bootstrap and ...
Abdel_DAS's user avatar
5 votes
1 answer
2k views

Boosting reduces bias when compared to what algorithm?

I am reading on bagging and boosting, and I understand how they both work (at least I think I do). I would like to talk in the context of decision tree ensembles as I think (not sure if correct) that ...
Slim Shady's user avatar
0 votes
0 answers
960 views

R Caret - Random Forest with repeated CV

I am currently struggling to explain the combination of bagging and cross-validation. I am aware of what each does separately but I have difficulties to explain how they are combined (if at all). For ...
Ru8zi9's user avatar
  • 1
4 votes
1 answer
1k views

Is sampling with replacement better than sampling without replacement?

This question is more specific to machine learning. Is sampling with replacement good for random forests because it leaves some out of bag samples for testing or is it because it creates datasets/...
Ixion Chowdhury's user avatar
0 votes
0 answers
45 views

RandomForest with VST data

I used the randomForest() function in R for bagging without a training data set to identify the important features characterizing the members of groups A and B (binary testing). I get the following ...
AennaPhD's user avatar
6 votes
1 answer
207 views

Why is the size of the feature bagging sample typically the square root of the total predictor set size?

Applying feature sampling to each tree in the context of a Random Forest model, if the set of predictors is of size $p$, why does the size of the sample of predictors for each split typically be of ...
tempMC's user avatar
  • 61
1 vote
1 answer
335 views

Can we make bagging equivalent to random forest if we take max features in our own defined decision tree and pass it as estimator?

If in bagging we defined our own tuned decision tree as estimator with max features as parameter in that estimator rather than taking by default decision tree estimator available in BaggingClassifer ...
Raufur Khan's user avatar
8 votes
2 answers
590 views

Intutition of why Bootstrap aggregating reduces overfitting?

Can somebody give me a non-mathematical intuition why Bootstrap aggregating reduces overfitting? From my point of view, we are not providing any additional information, we are not really enlengthen ...
J3lackkyy's user avatar
  • 735
3 votes
2 answers
607 views

R Squared (OOB) and R Square from correlation of prediction of test set is different?

I'm using simulated data and fitting Random Forest model for regression on a training dataset. What is confusing me is that after running Random Forest, I got R Squared (OOB) equal to 0.14. But when I ...
Atiq Ur Rehman's user avatar
1 vote
1 answer
245 views

Why can't we sample without replacement for each tree in a random forest if the subsample size is large enough?

Usually if we have $n$ observations, for each tree with form a bootstrapped subsample of size $n$ with replacement. On googling it one common explanation I've seen is that with replacement sampling is ...
user9343456's user avatar
6 votes
2 answers
4k views

AdaBoost - why decision stumps instead of trees?

Since the original AdaBoost article it has been found out that boosting reduces both variance and bias in the classifier (in contrast to bagging, which only reduces variance). Original AdaBoost (and ...
qalis's user avatar
  • 238
3 votes
2 answers
419 views

Elements of Statistical Learning: Variance Reduction via Bagging for Random Forests

While Reading chapter 15 of ESL about Random Forests I had some confusion with this phrase: The idea in random forests (Algorithm 15.1) is to improve the variance reduction of bagging by reducing the ...
Christian Singer's user avatar
0 votes
1 answer
125 views

Is there a model, apart from Random Forest, which uses bagging?

What are the popular machine learning algorithms which make use of bagging? Are there any apart from random forest?
nyptop's user avatar
  • 23
1 vote
0 answers
42 views

Regarding bagging, boosting and the NFL theorem

According to my understanding, bagging and boosting work in the following way: Bagging: Combine several high-variance/low-bias models to produce an ensemble model with lower variance and equal bias. ...
Enk9456's user avatar
  • 33
1 vote
1 answer
37 views

Do tree methods like gradient boosting predict all iterations at once?

If I'm using a tree method (e.g GBM) and I have a time series hourly data, and I predict my target variable $y$ for the next 48 hours, do my predictions were made all at once, or does the second day ...
Numbermind's user avatar
2 votes
1 answer
619 views

How does bagging affect linear model assumptions?

Linear regression has assumptions. How does bagging affect model assumptions for linear regression? Also, should you build a bagged linear model with correlated and statistically significant variables ...
reeves's user avatar
  • 21
2 votes
0 answers
450 views

Performing Bagging with the Lasso

I asked this question at StackOverflow but I think it might be better suited for here because I'm hoping to understand the general idea of bagging. I'm trying to implement bagging in ...
AW27's user avatar
  • 253
1 vote
1 answer
173 views

bias-variance tradeoff is conflict to bagging

I first give the notations of bias-variance tradeoff: sample set: $D = \{(x_1,y_1),\cdots,(x_N,y_N)\}.$ relation between $y$ and $x:$ $y = f(x) + \epsilon.$ here $f(x)$ is a deterministic function; $...
user6703592's user avatar
  • 1,345
4 votes
2 answers
796 views

How many neglected samples when drawn with replacement? (bagging)

I learned a while ago about an interesting place that $e$ shows up in probability: if there are $n$ items and you sample $n$ times with replacement, you would expect that the fraction of samples that ...
efthimio's user avatar
  • 207
1 vote
0 answers
14 views

Is there a way to know which feature is causing a split or present in root or decision node of a decision tree in Random Forest using Python? [closed]

Let's say I have 100 features f1, f2, .... ,f100 and I want to know if f3, f4, f5 caused any split or were present in a root/decision node of any decision tree in a random forest. I have close to 200 ...
user_beginner's user avatar
1 vote
1 answer
477 views

What is the limit of the random forest (or bagging) estimator?

I am looking for a proof or intuition as to why the absolute limit of a random forest estimator is the expectancy of a single tree (see citation below), i.e: $$ \hat{f}_{rf}(x) = \lim_{B \to \infty}[\...
PaulG's user avatar
  • 1,297
1 vote
0 answers
45 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
DiamondDust's user avatar
2 votes
1 answer
912 views

Is n_estimators in BaggingRegressor() the number of trees or data subsets?

I'm learning about using the BaggingRegressor() from scikit learn (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html) Its <...
DN1's user avatar
  • 111