Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

Filter by
Sorted by
Tagged with
0 votes
0 answers
39 views

Bagging with SVM and Neural Networks in R with caret

I am fairly new to the bagging technique and Caret's bagControl() as well as bag() and am currently trying to build an ensemble ...
user avatar
  • 1
0 votes
0 answers
11 views

Bias, Variance and Bagging and what it means in relation to representational complexity

Re-looking at some basic ML algorithms bagging comes up along with the Bias-Variance trade off. I am confused on how bagging relates to representational capacity. Based upon the arguments I have read, ...
user avatar
0 votes
0 answers
15 views

Regression Trees

Which is better over the other two? Random Forest, Bagging, or Boosting the tree-based method? My understanding is, that even though all three have their own preferred requirements to perform better. ...
user avatar
  • 1
2 votes
1 answer
42 views

Does bagging work for OLS to improve prediction?

In Elements of Statistical Learning, section 8.7, the author states that The bagged estimate will differ from original estimate only when the latter is a nonlinear or adaptive function of the data I ...
user avatar
  • 101
0 votes
0 answers
39 views

Why does Random Forest perform worse than Bagging?

The 2 results I got for bagging and random forest are shown below. It seems that calculating mean MSE from bootstrapping also result in a lower mean MSE for bagging as compared to random forest. Is ...
user avatar
0 votes
3 answers
33 views

Why does test MSE always decrease with increasing training size (and decreasing test size)?

Context: I am trying to find the best predictive model for a dataset with 1000 observations. The problem is I am not sure what the best training and test size should be. So what I did was that I ran ...
user avatar
1 vote
1 answer
44 views

Random Forest - varying seed to quantify uncertainty

I would like to quantify the uncertainty of a Random Forest binary classifier. The idea that popped in my mind was to fit the Random Forest 100 times with different seeds. Computing the variance of ...
user avatar
0 votes
0 answers
12 views

Can I use the learning set as a test set for bootstrapped predictors?

This question relates to Leo Breiman's paper: Bagging Predictors from 1996. The author claims that if bagging is deployed, the original training set can be used to assess the performance of the ...
user avatar
1 vote
1 answer
20 views

How does the performance of bagging depend on instability?

This question relates to Leo Breiman's paper: Bagging Predictors from 1996. Assuming that $\mathcal{L}$ denotes the training set and $\phi$ the predictor which depends on the training set and the ...
user avatar
1 vote
0 answers
19 views

Lowering OLS prediction error with bootstrapping/bagging - a free lunch?

From what I understand, bagging reduces variance of prediction for a model. Though OLS is on the "low variance" part of the spectrum, I wish to understand anyway the implications of ...
user avatar
  • 129
1 vote
0 answers
20 views

Bagging dependent data

Which are the possible caveats of using a Bagging algorithm (such as Random Forest), when data are not independent? Ensemble models usually exploit Bagging to reduce the variance by aggregating ...
user avatar
1 vote
1 answer
27 views

proof out of bag evaluation [duplicate]

In Geron's book "Hands-on Machine Learning with Scikit-Learn and Tensorflow" there is this sentence on page 187 "By default a BaggingClassifier samples m training instances with ...
user avatar
  • 155
0 votes
0 answers
19 views

Comparison between regression models based on trees

I'm solving a prediction problem in which I have an independent variables Y and 13 dependent variables which also are highly correlated. My dataset is composed by 124 observation for the train dataset ...
user avatar
  • 1
0 votes
1 answer
131 views

How to bootstrap time series data

I have this dataset that contains multiple series (50 products). My dataset has 50 products (50 columns). each column contains the monthly sales of a product. I recently learned about bootstrap and ...
user avatar
2 votes
1 answer
92 views

Boosting reduces bias when compared to what algorithm?

I am reading on bagging and boosting, and I understand how they both work (at least I think I do). I would like to talk in the context of decision tree ensembles as I think (not sure if correct) that ...
user avatar
0 votes
0 answers
124 views

Average Forecast between Bumping and Bagging without any Programming!

Applying your model to 5 bootstrap samples, you obtain the following results: 8,6,9,2,6 the average forecast between bumping and bagging is (round your result to 2 decimal places): I need help with ...
user avatar
0 votes
0 answers
80 views

Explain how bagging reduces Variance and Boosting reduces Bias mathematically? [duplicate]

I am unable to find an answer to this question, even in some famous books. I know a bit about bagging and boosting and also know that they reduce variance (overfitting) and bias respectively. But I am ...
user avatar
0 votes
0 answers
264 views

R Caret - Random Forest with repeated CV

I am currently struggling to explain the combination of bagging and cross-validation. I am aware of what each does separately but I have difficulties to explain how they are combined (if at all). For ...
user avatar
  • 1
4 votes
1 answer
491 views

Is sampling with replacement better than sampling without replacement?

This question is more specific to machine learning. Is sampling with replacement good for random forests because it leaves some out of bag samples for testing or is it because it creates datasets/...
user avatar
0 votes
0 answers
34 views

RandomForest with VST data

I used the randomForest() function in R for bagging without a training data set to identify the important features characterizing the members of groups A and B (binary testing). I get the following ...
user avatar
6 votes
1 answer
55 views

Why is the size of the feature bagging sample typically the square root of the total predictor set size?

Applying feature sampling to each tree in the context of a Random Forest model, if the set of predictors is of size $p$, why does the size of the sample of predictors for each split typically be of ...
user avatar
  • 61
0 votes
1 answer
95 views

Can we make bagging equivalent to random forest if we take max features in our own defined decision tree and pass it as estimator?

If in bagging we defined our own tuned decision tree as estimator with max features as parameter in that estimator rather than taking by default decision tree estimator available in BaggingClassifer ...
user avatar
7 votes
2 answers
167 views

Intutition of why Bootstrap aggregating reduces overfitting?

Can somebody give me a non-mathematical intuition why Bootstrap aggregating reduces overfitting? From my point of view, we are not providing any additional information, we are not really enlengthen ...
user avatar
  • 535
0 votes
0 answers
11 views

Why does my Bagging Ensemble model overfit? [duplicate]

I implemented a Bagging Ensemble model in sklearn, but the cross_validate method always returns a perfect train score. And here are my results Is this behaviour normal or am I doing something wrong ...
user avatar
  • 1
0 votes
0 answers
44 views

Boosting models performing worse than bagging models

I've noticed that, sometimes, models that are based on boosting, such as Gradient Boosting, show worse performance than pretty similar models, but based on bagging. For example, Random Forest (bagging ...
user avatar
1 vote
1 answer
109 views

Why can't we sample without replacement for each tree in a random forest if the subsample size is large enough?

Usually if we have $n$ observations, for each tree with form a bootstrapped subsample of size $n$ with replacement. On googling it one common explanation I've seen is that with replacement sampling is ...
user avatar
2 votes
2 answers
1k views

AdaBoost - why decision stumps instead of trees?

Since the original AdaBoost article it has been found out that boosting reduces both variance and bias in the classifier (in contrast to bagging, which only reduces variance). Original AdaBoost (and ...
user avatar
  • 179
2 votes
2 answers
92 views

Elements of Statistical Learning: Variance Reduction via Bagging for Random Forests

While Reading chapter 15 of ESL about Random Forests I had some confusion with this phrase: The idea in random forests (Algorithm 15.1) is to improve the variance reduction of bagging by reducing the ...
user avatar
0 votes
1 answer
54 views

Is there a model, apart from Random Forest, which uses bagging?

What are the popular machine learning algorithms which make use of bagging? Are there any apart from random forest?
user avatar
  • 23
1 vote
0 answers
32 views

Regarding bagging, boosting and the NFL theorem

According to my understanding, bagging and boosting work in the following way: Bagging: Combine several high-variance/low-bias models to produce an ensemble model with lower variance and equal bias. ...
user avatar
  • 23
0 votes
1 answer
29 views

Do tree methods like gradient boosting predict all iterations at once?

If I'm using a tree method (e.g GBM) and I have a time series hourly data, and I predict my target variable $y$ for the next 48 hours, do my predictions were made all at once, or does the second day ...
user avatar
0 votes
1 answer
81 views

How does bagging effect linear model assumptions?

Linear regression has model assumptions. How does bagging effect model assumptions for linear regression? Also, should you build a bagged linear model with correlated and statistically significant ...
user avatar
  • 1
2 votes
0 answers
140 views

Performing Bagging with the Lasso

I asked this question at StackOverflow but I think it might be better suited for here because I'm hoping to understand the general idea of bagging. I'm trying to implement bagging in ...
user avatar
  • 233
0 votes
1 answer
62 views

bias-variance tradeoff is conflict to bagging

I first give the notations of bias-variance tradeoff: sample set: $D = \{(x_1,y_1),\cdots,(x_N,y_N)\}.$ relation between $y$ and $x:$ $y = f(x) + \epsilon.$ here $f(x)$ is a deterministic function; $...
user avatar
2 votes
2 answers
265 views

How many neglected samples when drawn with replacement? (bagging)

I learned a while ago about an interesting place that $e$ shows up in probability: if there are $n$ items and you sample $n$ times with replacement, you would expect that the fraction of samples that ...
user avatar
  • 129
1 vote
0 answers
11 views

Is there a way to know which feature is causing a split or present in root or decision node of a decision tree in Random Forest using Python? [closed]

Let's say I have 100 features f1, f2, .... ,f100 and I want to know if f3, f4, f5 caused any split or were present in a root/decision node of any decision tree in a random forest. I have close to 200 ...
user avatar
1 vote
1 answer
242 views

What is the limit of the random forest (or bagging) estimator?

I am looking for a proof or intuition as to why the absolute limit of a random forest estimator is the expectancy of a single tree (see citation below), i.e: $$ \hat{f}_{rf}(x) = \lim_{B \to \infty}[\...
user avatar
  • 883
1 vote
0 answers
29 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
user avatar
2 votes
1 answer
221 views

Is n_estimators in BaggingRegressor() the number of trees or data subsets?

I'm learning about using the BaggingRegressor() from scikit learn (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html) Its <...
user avatar
  • 69
2 votes
1 answer
728 views

How does bagging reduce variance?

I read this answer. Was still unable to understand how bagging reduces variance. Is there any other way to explain it mathematically to a newbie ? Edit Can anybody explain me this excerpt from the ...
user avatar
  • 119
1 vote
1 answer
74 views

Bagging models with different metaparameters versus cross validation?

If we have a model with a metaparameter C, the usual way to tune this parameter is via cross-validation (or building a validation set). The generalization error of the model with the optimal is ...
user avatar
  • 663
2 votes
1 answer
260 views

Overfit in aggregated models: boosting versus simple bagging

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the ...
user avatar
  • 663
3 votes
1 answer
529 views

What is the purpose of using duplicated data in resampling techniques (e.g., bagging/bootstrapping)?

With bootstrapping and bagging, we resample from the dataset and end up building a model or estimating some sample statistic using the sampled data, which typically consists of at least $33\%$ ...
user avatar
  • 887
1 vote
1 answer
21 views

Some ambiguities about used training dataset in bagged trees

I have some ambiguity about dividing training dataset in Bagging tree. In fact I have found in this article About Decision Tree Ensembles- Bagging That : the idea is to create several subsets of data ...
user avatar
  • 125
1 vote
2 answers
169 views

Is boosting and bagging only relevant in the context of decision trees?

In the documents I've seen on boosting and bagging, it seems that they're always talked about in the context of decision trees. What are some other methods in which the two are applicable?
user avatar
  • 563
0 votes
0 answers
30 views

Are predictions generated from bootstrap resampling normally distributed?

If I resample a dataset hundreds of times with replacement (bootstrap replications), fit some Model A on each bootstrap sample, and generate predictions (time series) from each fitted model, can I ...
user avatar
  • 3,067
3 votes
2 answers
460 views

Bagging - Size of the aggregate bags?

I'm reading up on bagging (boostrap aggregation), and several sources seem to state that the size of the bags (consist of random sampling from our training set with replacement) is typically around 63%...
user avatar
  • 563
2 votes
1 answer
27 views

Bagging sample probability

I know that the probability that an observation does not appear in the sample is $\left(1 - \frac{\ 1}{n} \right)^n ≈ \left(\frac{1}{e}\right) $ (when n is large). My question is how to find the ...
user avatar
  • 27
0 votes
0 answers
719 views

Time complexity of bagging and random forest

Most sources and books state that time complexity of a single decision tree for n points and d dimensions (features) is $O(d * n^2 * log(n))$, with clever caching and one time sorting it’s $O(d * n * ...
user avatar
  • 179
2 votes
1 answer
153 views

Why are boosted trees difficult to interpret?

I might not fully understand the topic but: Bootstrapped/bagged trees are difficult to interpret because the decisions are made from averaging the prediction of possibly hundreds of trees (ensemble). ...
user avatar
  • 133