Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Bagging variance between train and test

I’m doing an experiment where I have 5k data points in a train dataset, I’m sampling with replacement several samples of 5k data points, and using unpruned trees to create a Model for each sample, ...
0
votes
1answer
41 views

Bagging results from multiple k-fold cross-validation runs to estimate performance

Given a binary classification problem and a model construction algorithm, suppose I run, as an example, 5 fold (stratified) cross-validation 4 times. I can think of these as 5x4 = 20 training and 20 ...
0
votes
0answers
13 views

Should we include “YEAR” column in time series data for any tree based algorithm? [duplicate]

I've a time series data having day, month, year and other columns. If I include year column during building random forest, then how can I predict for the new year. As the year passed won't be repeated ...
1
vote
0answers
14 views

Trying to improve the generalization ability of decision tree with bagging or random forest

We are fitting a regression tree on a sample of about 5000 observations with 5 predictors. The data stems from 2 sources. There are reasons to suspect that the tree will not have exaggerated ...
0
votes
1answer
27 views

Machine Learning - Bagging

I am very new to Machine Learning I know i should use Decision Tree for bagging model because of its high variance. why cant we use any other algorithms for bagging ? if we can use any other ...
2
votes
1answer
117 views

Why is bagging stable classifiers not a good idea?

Citing Bagging Predictors, Section 4 (emphasis mine): Let each $(y, \mathbf{x})$ case in $\mathcal{L}$ be independently drawn from the distribution $P$. Suppose $y$ is numerical and $\phi(\mathbf{x}...
1
vote
1answer
33 views

why does random forest trees need to be deeper than gradient boosting trees

in Elements of Statistical Learning chapter 15. Random Forest, we see authors' note on RF v.s. GBT. One of them is that at 1000 terms, GBM depth 4 has smaller error than RF depth 6. Also we notice RF ...
0
votes
2answers
66 views

Using bagging and random forests together

I was looking at a kernel implementation (for text classification) and the following piece of code got me a little bit confused (I removed part of the features - in order to keep it light - as most of ...
2
votes
1answer
71 views
1
vote
0answers
14 views

Bagging Classifiers

In Bootstrap aggregating, instances that have not been drawn (by resampling with replacement), e.g. end up for example in the test set. However, in the bootstrap sample not every instance is unique. ...
1
vote
0answers
32 views

Does it make sense to use bagging with least square regression? if so, why?

We know that bagging can reduce the variance of the estimator, while keep the bias around the same. If we use bagging to ensemble multiple least square regressors, then are we going to reduce the ...
1
vote
1answer
105 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
6
votes
1answer
167 views

Variance of average of $n$ correlated random variables

Reading about deep leaning, I came across the following formula. $$ \mbox{var} \left( \frac{1}{n} \sum_{i=1}^{n} X_i \right) = \rho \sigma^2 + \frac{1-\rho}{n} \sigma^2 $$ where $X_1, \dots, X_n$ ...
0
votes
1answer
33 views

Is Gini index actively involved in splitting a Random Forest node?

Since there are many references that a RF uses a slightly different approach on splitting a node in comparison to Vanilla Bagging. Does Gini index play an active role in the split or it's just another ...
0
votes
1answer
70 views

Is bagging involved in the split of node of a tree of a Random Forest?

I know that using Bagging method in a RF, implies that the subset we give to the root node of each tree, has randomly selected Features and Attributes. I also know that during the split of a node ...
-1
votes
1answer
64 views

Parallel Bagging in supervised learning

How can we parallelize Bootstrap aggregation, a.k.a Bagging, i.e. train all classifiers at once?
0
votes
0answers
25 views

What is the efficient algorithm among Bagged Trees, Neural Network (TensorFlow) and C4.5?

I am performing a classification process over a collection of signals where each signal has 12 parameter. I need to predict my class/ label using those 12 features, the classes are 5 from 0 to 4 (It ...
3
votes
1answer
1k views

How can we explain the fact that “Bagging reduces the variance while retaining the bias” mathematically?

I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias". What is the mathematically principle behind this intution? I checked with few experts ...
12
votes
5answers
3k views

Random Forest and Decision Tree Algorithm

A random forest is a collection of decision trees following the bagging concept. When we move from one decision tree to the next decision tree then how does the information learned by last decision ...
1
vote
1answer
102 views

Aggregating method used by BaggingClassifier and BaggingRegressor?

What aggregating method is used by BaggingClassifier and BaggingRegressor in sklearn? Do they use soft voting or hard voting or averaging? The sklearn docs don't seem to clearly specify it.
4
votes
1answer
656 views

Why does bagging increase bias?

In machine learning, why does bagging increase bias? I've read that using less data would lead to a worse estimate of the parameters, but isn't the expected value of the parameter constant regardless ...
15
votes
1answer
1k views

Boosting AND Bagging Trees (XGBoost, LightGBM)

There are many blog posts, YouTube videos, etc. about the ideas of bagging or boosting trees. My general understanding is that the pseudo code for each is: Bagging: Take N random samples of x% of ...
2
votes
1answer
288 views

Random Forest: Number of Bags vs. Number of Trees

What are the differences between the number of bags and the number of trees in a Random Forest? For what I knew number of bags = number of tress in RF. However, in certain packages such as RankLib, ...
1
vote
0answers
239 views

What is the relation between minimum instances per node and max depth?

In bagging and boosting models like random forest and xgboost we have hyper-parameters like minimum instances per node and max depth. If max depth is high the minimum instances per node will be less ...
3
votes
2answers
1k views

What are advantages of random forests vs using bagging with other classifiers?

I'm studying Random Forests, but after reviewing the methods I got the following line of reasoning: I feel like the big advantage of random forests happens in the bagging process where nearly ...
0
votes
1answer
27 views

Why does creating training sets with replacement lead to better performance?

Pasting generally suffers from lower performance than bagging, because it training sets without replacement. Why does creating training sets with replacement lead to better performance?
1
vote
1answer
23 views

Sample size for designing a study which creates a classifier

I have to plan a study in which I will have to create a classifier. The output variable is a binary with an estimated proportion of value 1 in the overall population of interest to be 0.10 (and ...
8
votes
1answer
1k views

Can I combine many gradient boosting trees using bagging technique

Based on Gradient Boosting Tree vs Random Forest . GBDT and RF using different strategy to tackle bias and variance. My question is that can I resample dataset (with replacement) to train multiple ...
3
votes
0answers
36 views

Bagging of models with link functions

I'm trying to predict proportion data, and I've got a small dataset (~4000), so holding out a test and validation set isn't practical. However, bagging is practical because the cost of training isn't ...
14
votes
1answer
309 views

What bagging algorithms are worthy successors to Random Forest?

For boosting algorithms, I would say that they evolved pretty well. In early 1995 AdaBoost was introduced, then after some time it was Gradient Boosting Machine (GBM). Recently, around 2015 XGBoost ...
2
votes
0answers
94 views

Are boosted trees always better than bagging trees?

I've observed this on quite a few different types of datasets, GBM or XGBOOST always perform better than ...
3
votes
2answers
339 views

Bagging vs pasting: bias-variance tradeoff

In the Hands-On ML with Scikit-Learn book, it states that, ...bagging ends up with a slightly higher bias than pasting, but... the ensemble's variance is reduced. I am a bit confused about this ...
2
votes
1answer
616 views

Combining bagging and stacking, with and without clusters and heteroskedasticity

Question 1: Start with the classing case of bagging, say in random forest. Fit $B$ trees to bootstrap samples of the data. Average the predictions of the $B$ trees to form a final prediction. ...
2
votes
3answers
632 views

What base classifiers to use with bagging?

I am working on a specific implementation of bootstrap aggregating (bagging). I want to see how well this bagging works for different base classifiers. But so far, the decision tree seems to be the ...
2
votes
1answer
431 views

K-fold cross-bagging?

Typically when one does cross-validation, one fits $K$ models of the same form over a grid of a hyperparameter. One selects the hyperparameter that minimizes the prediction error out-of-sample ...
4
votes
2answers
4k views

How is bagging different from cross-validation?

How is bagging different from cross-validation? Can a data set having 300 examples can be 100 bagged and would it be helpful at all?
-1
votes
1answer
298 views

How to tune parameters of Bagging in R? [closed]

I was wondering when we use the Bagging to do the classification, what parameters can be tuned and can we use the cross-validation to tune it? In the Bagging function in R, it says we can use the ...
7
votes
1answer
1k views

Is it pointless to use Bagging with nearest neighbor classifiers?

In page 485 of the book [1], it is noted that "it is pointless to bag nearest-neighbor classifiers because their output changes very little if the training data is perturbed by sampling". This is ...
0
votes
0answers
33 views

Forecasting methods to work globally on a series of datasets

Forecasting a numeric time-series is providing an estimator $\hat y(t+1)$ of $y(t+1)$ computed from $(y(1),y(2)...y(t))$. You want it to be close to $y(t+1)$ (say in terms of squared distance). Now ...
1
vote
0answers
124 views

Central limit theorem in Bagging

I am in process of trying to understand the statistical theory behind Machine learning. I came across the fact that central limit theorem plays a key role in the Bagging algorithm (in ML). I searched ...
5
votes
1answer
2k views

Why is tree correlation a problem when working with bagging?

I'm reading ISLR and I don't understand what is the problem that random forests solve; what problems does tree correlation cause when using bagging?
1
vote
0answers
45 views

Finding good sub-populations for modeling

I've built a (deep learning) model for a binary classification problem. For this problem I'm interested in sub-dividing my data set to maximize performance (as measured by precision) by essentially ...
10
votes
2answers
4k views

Why does a bagged tree / random forest tree have higher bias than a single decision tree?

If we consider a full grown decision tree (i.e. an unpruned decision tree) it has high variance and low bias. Bagging and Random Forests use these high variance models and aggregate them in order to ...
5
votes
2answers
14k views

Is cross validation unnecessary for Random Forest?

Is it fair to say Cross Validation (k-fold or otherwise) is unnecessary for Random Forest? I've read that is the case because we can look at out-of-bag performance metrics, and these are doing the ...
1
vote
1answer
79 views

Can I re-weight the a training data set to make it more relevant in the selection of algorithms for use on a slightly different data set?

Suppose $(x_i,y_i)$ constitutes supervision data. Assume a collection of statistical methods which could be trained on one subset and ranked by their accuracy on another subset of this data (according ...
2
votes
0answers
241 views

Why bagging algorithm doesn't influence bias

I am learning the bagging algorithm and I have realized that bagging can reduce the variance, but I am confused when I read that bagging doesn't influence the bias. Here is a picture in the slides ...
1
vote
1answer
491 views

Why would a BaggingRegressor only use a subset of samples and features during fitting?

I am working with a BaggingRegressor from sklearn and am having difficulty understanding what the purpose/effect of max_features and ...
5
votes
1answer
3k views

XGBoost feature subsampling

I have a dataset with ~30k samples and 35 features (after feature selection; these seem to be the most important features for this dataset and they have low correlation between each other). After ...
1
vote
0answers
63 views

400 features and 100 classes using weka

I am working on a classification problem, where I have 400 features(all are numeric), and 100 classes and I have 26,000 examples for training. In my project I am using Weka and I have tried different ...
1
vote
0answers
94 views

Bagging for continuous variable

I am using bagging approach in R to predict average Ozone data. I have a confusion that, can I use bagging method to predict average Ozone? Since it is continuous variable. I read some articles where ...