Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

Filter by
Sorted by
Tagged with
3
votes
2answers
157 views

Bagging - Size of the aggregate bags?

I'm reading up on bagging (boostrap aggregation), and several sources seem to state that the size of the bags (consist of random sampling from our training set with replacement) is typically around 63%...
3
votes
1answer
1k views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
3
votes
1answer
971 views

Combining bagging and stacking, with and without clusters and heteroskedasticity

Question 1: Start with the classing case of bagging, say in random forest. Fit $B$ trees to bootstrap samples of the data. Average the predictions of the $B$ trees to form a final prediction. ...
3
votes
1answer
532 views

Random Forest: Number of Bags vs. Number of Trees

What are the differences between the number of bags and the number of trees in a Random Forest? For what I knew number of bags = number of tress in RF. However, in certain packages such as RankLib, ...
3
votes
1answer
246 views

Bias in bagged decision tree estimates of probability (classification)

As far as I understand it, random forests are as biased as any tree in the forest. Bagging a non-random forest (using all available variables at any given split) is unbiased -- bias in random forests ...
3
votes
1answer
2k views

Bias-variance decomposition with sklearn BaggingRegressor

There is an example given on the Scikit-Learn site that compares the bias-variance decomposition of the rmse of a single SVR model against a bagging ensemble. Unfortunately, the data is being ...
3
votes
2answers
116 views

How many neglected samples when drawn with replacement? (bagging)

I learned a while ago about an interesting place that $e$ shows up in probability: if there are $n$ items and you sample $n$ times with replacement, you would expect that the fraction of samples that ...
3
votes
0answers
40 views

Bagging of models with link functions

I'm trying to predict proportion data, and I've got a small dataset (~4000), so holding out a test and validation set isn't practical. However, bagging is practical because the cost of training isn't ...
3
votes
0answers
129 views

Are boosted trees always better than bagging trees?

I've observed this on quite a few different types of datasets, GBM or XGBOOST always perform better than ...
3
votes
0answers
442 views

State of the art for bagging/model averaging? [closed]

If I estimate a collection of models predicting $Y$ by $\hat{Y}$, which methods are out there to combine these forecasts? Which methods work well/best (and why?) to improve prediction accuracy? My ...
2
votes
1answer
117 views

Why are boosted trees difficult to interpret?

I might not fully understand the topic but: Bootstrapped/bagged trees are difficult to interpret because the decisions are made from averaging the prediction of possibly hundreds of trees (ensemble). ...
2
votes
1answer
217 views

Is there a well-defined class of ensemble methods?

Ensemble methodology's main aim is to somehow aggregate or summarize estimates from multiple models. In some cases this is aggregating different bootstrap estimates or Monte Carlo estimates, but ...
2
votes
2answers
35 views

Elements of Statistical Learning: Variance Reduction via Bagging for Random Forests

While Reading chapter 15 of ESL about Random Forests I had some confusion with this phrase: The idea in random forests (Algorithm 15.1) is to improve the variance reduction of bagging by reducing the ...
2
votes
1answer
1k views

Use of a bagging model or feature engineering?

As a pet project, I have been learning some data analysis and machine learning skills (mainly text analytics) with the Analytics Edge course on edX. I decided to put some of my new skills at use ...
2
votes
1answer
17 views

Bagging sample probability

I know that the probability that an observation does not appear in the sample is $\left(1 - \frac{\ 1}{n} \right)^n ≈ \left(\frac{1}{e}\right) $ (when n is large). My question is how to find the ...
2
votes
1answer
113 views

Why is the error rate from bagging trees much higher than that from a single tree?

I'm running the classification method Bagging Tree (Bootstrap Aggregation) and compare the misclassification error rate with one from one single tree. We expect that the result from bagging tree is ...
2
votes
1answer
521 views

What is the advantage of bagging for k-NN?

Is there an advantage to use bagging with k-NN? I constantly get better performances while doing it. Can it be that it is because of resampling with repeated instances, which will therefore be ...
2
votes
1answer
889 views

What happens when Bagging does not have a majority vote?

I have a question regarding the bagging technique used in ensemble learning. Let's assume I have 6 classifiers which could classify a response variable which has 3 finite categories(...
2
votes
1answer
923 views

KNN with bagging in R

How to implement bagging with KNN using R in order to reduce the variability? This is the R code that I use for KNN ...
2
votes
1answer
2k views

Out-Of-Bag estimate in scikit-learn

I am using a bagging model from the Python Scikit-Learn module: ...
2
votes
2answers
3k views

Is highly correlated factors in a prediction model a problem?

I have built a logistic regression model with two or more highly correlated factors. I did this by doing a bagging procedure. In my understanding having highly correlated factors in a prediction ...
2
votes
1answer
578 views

Building the dataset for Random Forest training procedure

I should use the bagging (bootstrap aggregating) technique in order to train a random forest classifier. I read here the description of this learning technique, but I have not figured out how I ...
2
votes
1answer
103 views

Is n_estimators in BaggingRegressor() the number of trees or data subsets?

I'm learning about using the BaggingRegressor() from scikit learn (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html) Its <...
2
votes
1answer
344 views

How does bagging reduce variance?

I read this answer. Was still unable to understand how bagging reduces variance. Is there any other way to explain it mathematically to a newbie ? Edit Can anybody explain me this excerpt from the ...
2
votes
1answer
146 views

Overfit in aggregated models: boosting versus simple bagging

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the ...
2
votes
1answer
288 views

Bagging vs downbagging

I am having difficulties in understanding the difference between bagging and downbagging. I understand that: Downbagging is an extension of bagging where downsampling is used. In downbagging, the ...
2
votes
1answer
398 views

Too many bagging estimators?

I am bagging 20 SVMs using the full training set. I have found the best SVM params using grid search. The validation performance is quite good, but performance on the training set is disappointing. ...
2
votes
0answers
53 views

Performing Bagging with the Lasso

I asked this question at StackOverflow but I think it might be better suited for here because I'm hoping to understand the general idea of bagging. I'm trying to implement bagging in ...
2
votes
0answers
9 views

Random forest and bagging: if both generate deep, non-pruned trees then what's the difference?

I understand that random forest only considers a subset of predictors at each node, but if you are generating deep trees then you'll eventually include all predictors in your model, the only ...
2
votes
0answers
167 views

Central limit theorem in Bagging

I am in process of trying to understand the statistical theory behind Machine learning. I came across the fact that central limit theorem plays a key role in the Bagging algorithm (in ML). I searched ...
2
votes
0answers
281 views

Why bagging algorithm doesn't influence bias

I am learning the bagging algorithm and I have realized that bagging can reduce the variance, but I am confused when I read that bagging doesn't influence the bias. Here is a picture in the slides ...
2
votes
0answers
1k views

Strategy for finding optimal bagging parameters

I am using a BaggingClassifier of SVMs in sklearn. What is the best strategy for finding optimal parameters, using my training/vaildation data? When using the full dataset, I can use grid search to ...
2
votes
0answers
399 views

Confidence interval for Bagging method in R

Here is the code which implements bagging that I copied from net (http://www.r-bloggers.com/improve-predictive-performance-in-r-with-bagging/) : ...
2
votes
0answers
1k views

cross validation vs bagging

I understand the ins and outs of the processes of both cross validation (partition the data set evenly, train on k-1 partitions, blah blah blah) and bagging (train M models composed of n observations ...
2
votes
0answers
3k views

What is the difference between oob (out of bag) error and (1 - accuracy) in RandomForest?

In a Random Forest, I know that the Out Of Bag Error is described as the fraction of number incorrect classifications over number of out of bag samples. Accuracy is defined as the number of correct ...
2
votes
0answers
120 views

Adjust p-values for multiple comparisons

I want to evaluate if a proposed modification M* to a base classifier M is better in terms of accuracy. Both, the base classifiers M and their respective modifications M* are tested on N datasets. To ...
1
vote
1answer
3k views

How bagging on CART (RPART) is different from CART with cross validation?

I am wondering is there any difference between the following two algorithms: RPART (Recursive partitioning) in R, with cross-validation (xval = 10, default) Bagging on RPART In the first case, Rpart ...
1
vote
1answer
108 views

Can I re-weight the a training data set to make it more relevant in the selection of algorithms for use on a slightly different data set?

Suppose $(x_i,y_i)$ constitutes supervision data. Assume a collection of statistical methods which could be trained on one subset and ranked by their accuracy on another subset of this data (according ...
1
vote
1answer
462 views

How small can a subsample in bagging be before performance degrades severely?

If I want to perform bagging, would subsamples with sizes of 0.1% of the actual data be appropriate? The reason I want to do so is because my actual data set is very large in the tens of millions.
1
vote
1answer
14 views

Why can't we sample without replacement for each tree in a random forest if the subsample size is large enough?

Usually if we have $n$ observations, for each tree with form a bootstrapped subsample of size $n$ with replacement. On googling it one common explanation I've seen is that with replacement sampling is ...
1
vote
2answers
154 views

AdaBoost - why decision stumps instead of trees?

Since the original AdaBoost article it has been found out that boosting reduces both variance and bias in the classifier (in contrast to bagging, which only reduces variance). Original AdaBoost (and ...
1
vote
1answer
67 views

Bagging models with different metaparameters versus cross validation?

If we have a model with a metaparameter C, the usual way to tune this parameter is via cross-validation (or building a validation set). The generalization error of the model with the optimal is ...
1
vote
1answer
450 views

why does random forest trees need to be deeper than gradient boosting trees

in Elements of Statistical Learning chapter 15. Random Forest, we see authors' note on RF v.s. GBT. One of them is that at 1000 terms, GBM depth 4 has smaller error than RF depth 6. Also we notice RF ...
1
vote
1answer
62 views

Sample size for designing a study which creates a classifier

I have to plan a study in which I will have to create a classifier. The output variable is a binary with an estimated proportion of value 1 in the overall population of interest to be 0.10 (and ...
1
vote
1answer
575 views

Why would a BaggingRegressor only use a subset of samples and features during fitting?

I am working with a BaggingRegressor from sklearn and am having difficulty understanding what the purpose/effect of max_features and ...
1
vote
1answer
120 views

What is the limit of the random forest (or bagging) estimator?

I am looking for a proof or intuition as to why the absolute limit of a random forest estimator is the expectancy of a single tree (see citation below), i.e: $$ \hat{f}_{rf}(x) = \lim_{B \to \infty}[\...
1
vote
1answer
21 views

Some ambiguities about used training dataset in bagged trees

I have some ambiguity about dividing training dataset in Bagging tree. In fact I have found in this article About Decision Tree Ensembles- Bagging That : the idea is to create several subsets of data ...
1
vote
1answer
113 views

Decision tree-based Bagging and Random Forest

I am new to Machine Learning and would like to know is it always true that decision tree-based bagging has worse predictive performance and is slower running time than random forest?
1
vote
1answer
134 views

Aggregating method used by BaggingClassifier and BaggingRegressor?

What aggregating method is used by BaggingClassifier and BaggingRegressor in sklearn? Do they use soft voting or hard voting or averaging? The sklearn docs don't seem to clearly specify it.
1
vote
1answer
1k views

K-fold cross-bagging?

Typically when one does cross-validation, one fits $K$ models of the same form over a grid of a hyperparameter. One selects the hyperparameter that minimizes the prediction error out-of-sample ...