Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

1
vote
0answers
22 views

Does it make sense to use bagging with least square regression? if so, why?

We know that bagging can reduce the variance of the estimator, while keep the bias around the same. If we use bagging to ensemble multiple least square regressors, then are we going to reduce the ...
1
vote
1answer
38 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
6
votes
1answer
142 views

Variance of average of $n$ correlated random variables

Reading about deep leaning, I came across the following formula. $$ \mbox{var} \left( \frac{1}{n} \sum_{i=1}^{n} X_i \right) = \rho \sigma^2 + \frac{1-\rho}{n} \sigma^2 $$ where $X_1, \dots, X_n$ ...
0
votes
1answer
22 views

Is Gini index actively involved in splitting a Random Forest node?

Since there are many references that a RF uses a slightly different approach on splitting a node in comparison to Vanilla Bagging. Does Gini index play an active role in the split or it's just another ...
0
votes
1answer
37 views

Is bagging involved in the split of node of a tree of a Random Forest?

I know that using Bagging method in a RF, implies that the subset we give to the root node of each tree, has randomly selected Features and Attributes. I also know that during the split of a node ...
-1
votes
1answer
30 views

Parallel Bagging in supervised learning

How can we parallelize Bootstrap aggregation, a.k.a Bagging, i.e. train all classifiers at once?
0
votes
0answers
23 views

What is the efficient algorithm among Bagged Trees, Neural Network (TensorFlow) and C4.5?

I am performing a classification process over a collection of signals where each signal has 12 parameter. I need to predict my class/ label using those 12 features, the classes are 5 from 0 to 4 (It ...
1
vote
1answer
134 views

How can we explain the fact that “Bagging reduces the variance while retaining the bias” mathematically?

I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias". What is the mathematically principle behind this intution? I checked with few experts ...
0
votes
0answers
18 views

Bagging - Strange result

Setting Training set size: 5200 Test set size: 1000 Question I'm currently working on a classifier which classifies images in a class (binary). My classifier uses a bagging approach (10 bags). I've ...
11
votes
5answers
2k views

Random Forest and Decision Tree Algorithm

A random forest is a collection of decision trees following the bagging concept. When we move from one decision tree to the next decision tree then how does the information learned by last decision ...
0
votes
1answer
63 views

Aggregating method used by BaggingClassifier and BaggingRegressor?

What aggregating method is used by BaggingClassifier and BaggingRegressor in sklearn? Do they use soft voting or hard voting or averaging? The sklearn docs don't seem to clearly specify it.
0
votes
0answers
15 views

ensemble of an ensemble in Scikit Learn

I am trying to get my head around ensemble learning and need some advice. Basically, my database contains a deterministic target variable and the feature variables are all stored as probability ...
3
votes
1answer
162 views

Why does bagging increase bias?

In machine learning, why does bagging increase bias? I've read that using less data would lead to a worse estimate of the parameters, but isn't the expected value of the parameter constant regardless ...
13
votes
1answer
614 views

Boosting AND Bagging Trees (XGBoost, LightGBM)

There are many blog posts, YouTube videos, etc. about the ideas of bagging or boosting trees. My general understanding is that the pseudo code for each is: Bagging: Take N random samples of x% of ...
2
votes
1answer
171 views

Random Forest: Number of Bags vs. Number of Trees

What are the differences between the number of bags and the number of trees in a Random Forest? For what I knew number of bags = number of tress in RF. However, in certain packages such as RankLib, ...
1
vote
0answers
96 views

What is the relation between minimum instances per node and max depth?

In bagging and boosting models like random forest and xgboost we have hyper-parameters like minimum instances per node and max depth. If max depth is high the minimum instances per node will be less ...
3
votes
1answer
822 views

What are advantages of random forests vs using bagging with other classifiers?

I'm studying Random Forests, but after reviewing the methods I got the following line of reasoning: I feel like the big advantage of random forests happens in the bagging process where nearly ...
0
votes
1answer
25 views

Why does creating training sets with replacement lead to better performance?

Pasting generally suffers from lower performance than bagging, because it training sets without replacement. Why does creating training sets with replacement lead to better performance?
1
vote
1answer
20 views

Sample size for designing a study which creates a classifier

I have to plan a study in which I will have to create a classifier. The output variable is a binary with an estimated proportion of value 1 in the overall population of interest to be 0.10 (and ...
6
votes
1answer
658 views

Can I combine many gradient boosting trees using bagging technique

Based on Gradient Boosting Tree vs Random Forest . GBDT and RF using different strategy to tackle bias and variance. My question is that can I resample dataset (with replacement) to train multiple ...
0
votes
0answers
32 views

Performing stacking classification

I have question regarding stacking classification. Just for the reference [The following kernel introduced to stacking classification method: https://www.kaggle.com/arthurtok/introduction-to-...
3
votes
0answers
35 views

Bagging of models with link functions

I'm trying to predict proportion data, and I've got a small dataset (~4000), so holding out a test and validation set isn't practical. However, bagging is practical because the cost of training isn't ...
0
votes
0answers
49 views

BaggingClassifier on predefined chunks of data

I have some data that can be splitted by k time periods. Is it possible to run BaggingClassifier on predefined ...
14
votes
1answer
258 views

What bagging algorithms are worthy successors to Random Forest?

For boosting algorithms, I would say that they evolved pretty well. In early 1995 AdaBoost was introduced, then after some time it was Gradient Boosting Machine (GBM). Recently, around 2015 XGBoost ...
2
votes
0answers
72 views

Are boosted trees always better than bagging trees?

I've observed this on quite a few different types of datasets, GBM or XGBOOST always perform better than ...
3
votes
2answers
219 views

Bagging vs pasting: bias-variance tradeoff

In the Hands-On ML with Scikit-Learn book, it states that, ...bagging ends up with a slightly higher bias than pasting, but... the ensemble's variance is reduced. I am a bit confused about this ...
2
votes
1answer
407 views

Combining bagging and stacking, with and without clusters and heteroskedasticity

Question 1: Start with the classing case of bagging, say in random forest. Fit $B$ trees to bootstrap samples of the data. Average the predictions of the $B$ trees to form a final prediction. ...
1
vote
3answers
332 views

What base classifiers to use with bagging?

I am working on a specific implementation of bootstrap aggregating (bagging). I want to see how well this bagging works for different base classifiers. But so far, the decision tree seems to be the ...
2
votes
1answer
284 views

K-fold cross-bagging?

Typically when one does cross-validation, one fits $K$ models of the same form over a grid of a hyperparameter. One selects the hyperparameter that minimizes the prediction error out-of-sample ...
3
votes
2answers
2k views

How is bagging different from cross-validation?

How is bagging different from cross-validation? Can a data set having 300 examples can be 100 bagged and would it be helpful at all?
-1
votes
1answer
250 views

How to tune parameters of Bagging in R? [closed]

I was wondering when we use the Bagging to do the classification, what parameters can be tuned and can we use the cross-validation to tune it? In the Bagging function in R, it says we can use the ...
5
votes
1answer
641 views

Is it pointless to use Bagging with nearest neighbor classifiers?

In page 485 of the book [1], it is noted that "it is pointless to bag nearest-neighbor classifiers because their output changes very little if the training data is perturbed by sampling". This is ...
0
votes
0answers
33 views

Forecasting methods to work globally on a series of datasets

Forecasting a numeric time-series is providing an estimator $\hat y(t+1)$ of $y(t+1)$ computed from $(y(1),y(2)...y(t))$. You want it to be close to $y(t+1)$ (say in terms of squared distance). Now ...
1
vote
0answers
98 views

Central limit theorem in Bagging

I am in process of trying to understand the statistical theory behind Machine learning. I came across the fact that central limit theorem plays a key role in the Bagging algorithm (in ML). I searched ...
4
votes
1answer
1k views

Why is tree correlation a problem when working with bagging?

I'm reading ISLR and I don't understand what is the problem that random forests solve; what problems does tree correlation cause when using bagging?
1
vote
0answers
44 views

Finding good sub-populations for modeling

I've built a (deep learning) model for a binary classification problem. For this problem I'm interested in sub-dividing my data set to maximize performance (as measured by precision) by essentially ...
10
votes
2answers
3k views

Why does a bagged tree / random forest tree have higher bias than a single decision tree?

If we consider a full grown decision tree (i.e. an unpruned decision tree) it has high variance and low bias. Bagging and Random Forests use these high variance models and aggregate them in order to ...
4
votes
2answers
10k views

Is cross validation unnecessary for Random Forest?

Is it fair to say Cross Validation (k-fold or otherwise) is unnecessary for Random Forest? I've read that is the case because we can look at out-of-bag performance metrics, and these are doing the ...
1
vote
1answer
67 views

Can I re-weight the a training data set to make it more relevant in the selection of algorithms for use on a slightly different data set?

Suppose $(x_i,y_i)$ constitutes supervision data. Assume a collection of statistical methods which could be trained on one subset and ranked by their accuracy on another subset of this data (according ...
3
votes
0answers
207 views

Why bagging algorithm doesn't influence bias

I am learning the bagging algorithm and I have realized that bagging can reduce the variance, but I am confused when I read that bagging doesn't influence the bias. Here is a picture in the slides ...
1
vote
1answer
407 views

Why would a BaggingRegressor only use a subset of samples and features during fitting?

I am working with a BaggingRegressor from sklearn and am having difficulty understanding what the purpose/effect of max_features and ...
4
votes
1answer
2k views

XGBoost feature subsampling

I have a dataset with ~30k samples and 35 features (after feature selection; these seem to be the most important features for this dataset and they have low correlation between each other). After ...
1
vote
0answers
55 views

400 features and 100 classes using weka

I am working on a classification problem, where I have 400 features(all are numeric), and 100 classes and I have 26,000 examples for training. In my project I am using Weka and I have tried different ...
1
vote
0answers
80 views

Bagging for continuous variable

I am using bagging approach in R to predict average Ozone data. I have a confusion that, can I use bagging method to predict average Ozone? Since it is continuous variable. I read some articles where ...
0
votes
0answers
1k views

Decision tree bagging performance in R

I am trying model a binary outcome variable. Logistic regression has already shown good results but I wanted to try to compare these results with some decision tree techniques. My training data set ...
1
vote
2answers
510 views

What is the relationship between bagging and XGBoost or Logistic Regression?

I am practicing classification with machine learning on a very large set of samples (about 20,000), where half of which are labelled training data and the other half are the testing data. There are 13 ...
1
vote
0answers
74 views

Class probability with Bagging

What is/are the standard methods to get the class probabilities with Bagging methods? For instance, with Random Forests, one can take the the class proportions at each terminal node. But generally ...
2
votes
1answer
374 views

What is the advantage of bagging for k-NN?

Is there an advantage to use bagging with k-NN? I constantly get better performances while doing it. Can it be that it is because of resampling with repeated instances, which will therefore be ...
0
votes
0answers
169 views

In Bagging, how does one train the base learner?

In Bagging (the grouping of predictive variables), I can set a decision tree, a neural network, and so forth as the base learner, since I can get m datasets randomly, and in each dataset, I can train ...
0
votes
0answers
235 views

The Appropriate p (number of predictors)/n (number of observations) Ratio in Tree-based Methods

I am dealing with a dataset in which there are 493 observations spanned over 30 predictors. My intention is to fit a model to make accurate predictions. It seems to me that the ratio $\frac{n}{p}$ ...