Questions tagged [bagging]

Bagging or bootstrap aggregation is a special case of model averaging. Given a standard training set bagging generates $m$ new training sets by bootstrapping, and then the results of using some training method on the $m$ generated data sets are averaged. Bagging can stabilize results from some unstable methods such as trees.

49 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
353 views

Is there an appropriate order to apply bagging and filter feature selection?

I'm training a (regression) learner on a $p \gg n$ problem, including bagging and filter feature selection (information gain). I'm in doubt though regarding the order of the procedures: Apply the ...
4
votes
0answers
344 views

Bag of Features / Visual Words + Locality Sensitive Hashing

PREMISE: I'm really new to Computer Vision/Image Processing and Machine Learning (luckily, I'm more expert on Information retrieval), so please be kind with this filthy peasant! :D MY APPLICATION: ...
4
votes
0answers
89 views

Is bagging a free lunch w.r.t. generalization error?

Since bagging seems to reduce variance without increasing the size of the hypothesis set (I think), is it fair to say that it does not increase the bias? Therefore, in terms of out-of-sample error, it ...
4
votes
2answers
598 views

Bagging vs pasting: bias-variance tradeoff

In the Hands-On ML with Scikit-Learn book, it states that, ...bagging ends up with a slightly higher bias than pasting, but... the ensemble's variance is reduced. I am a bit confused about this ...
3
votes
0answers
38 views

Bagging of models with link functions

I'm trying to predict proportion data, and I've got a small dataset (~4000), so holding out a test and validation set isn't practical. However, bagging is practical because the cost of training isn't ...
3
votes
0answers
120 views

Are boosted trees always better than bagging trees?

I've observed this on quite a few different types of datasets, GBM or XGBOOST always perform better than ...
2
votes
0answers
36 views

Performing Bagging with the Lasso

I asked this question at StackOverflow but I think it might be better suited for here because I'm hoping to understand the general idea of bagging. I'm trying to implement bagging in ...
2
votes
0answers
9 views

Random forest and bagging: if both generate deep, non-pruned trees then what's the difference?

I understand that random forest only considers a subset of predictors at each node, but if you are generating deep trees then you'll eventually include all predictors in your model, the only ...
2
votes
0answers
158 views

Central limit theorem in Bagging

I am in process of trying to understand the statistical theory behind Machine learning. I came across the fact that central limit theorem plays a key role in the Bagging algorithm (in ML). I searched ...
2
votes
0answers
275 views

Why bagging algorithm doesn't influence bias

I am learning the bagging algorithm and I have realized that bagging can reduce the variance, but I am confused when I read that bagging doesn't influence the bias. Here is a picture in the slides ...
2
votes
0answers
1k views

Strategy for finding optimal bagging parameters

I am using a BaggingClassifier of SVMs in sklearn. What is the best strategy for finding optimal parameters, using my training/vaildation data? When using the full dataset, I can use grid search to ...
2
votes
0answers
39 views

Are there simple memory-efficient ways to do multi-instance learning?

At the moment, I'm simply using mean of the features in all the instances in a bag to represent a given bag. I've also tried using min/max, gmean and hmean, but didn't get any better results. Are ...
2
votes
0answers
365 views

Confidence interval for Bagging method in R

Here is the code which implements bagging that I copied from net (http://www.r-bloggers.com/improve-predictive-performance-in-r-with-bagging/) : ...
2
votes
0answers
976 views

cross validation vs bagging

I understand the ins and outs of the processes of both cross validation (partition the data set evenly, train on k-1 partitions, blah blah blah) and bagging (train M models composed of n observations ...
2
votes
0answers
3k views

What is the difference between oob (out of bag) error and (1 - accuracy) in RandomForest?

In a Random Forest, I know that the Out Of Bag Error is described as the fraction of number incorrect classifications over number of out of bag samples. Accuracy is defined as the number of correct ...
2
votes
0answers
118 views

Adjust p-values for multiple comparisons

I want to evaluate if a proposed modification M* to a base classifier M is better in terms of accuracy. Both, the base classifiers M and their respective modifications M* are tested on N datasets. To ...
1
vote
0answers
16 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
1
vote
0answers
26 views

Why do correlated decision trees increase bias?

To decrease variance in decision trees often, bagging is used to create several decision trees. This has better accuracy than one tree since the decrease in variance outweighs the increase in bias. ...
1
vote
0answers
25 views

Trying to improve the generalization ability of decision tree with bagging or random forest

We are fitting a regression tree on a sample of about 5000 observations with 5 predictors. The data stems from 2 sources. There are reasons to suspect that the tree will not have exaggerated ...
1
vote
0answers
17 views

Bagging Classifiers

In Bootstrap aggregating, instances that have not been drawn (by resampling with replacement), e.g. end up for example in the test set. However, in the bootstrap sample not every instance is unique. ...
1
vote
0answers
49 views

Does it make sense to use bagging with least square regression? if so, why?

We know that bagging can reduce the variance of the estimator, while keep the bias around the same. If we use bagging to ensemble multiple least square regressors, then are we going to reduce the ...
1
vote
0answers
439 views

What is the relation between minimum instances per node and max depth?

In bagging and boosting models like random forest and xgboost we have hyper-parameters like minimum instances per node and max depth. If max depth is high the minimum instances per node will be less ...
1
vote
0answers
45 views

Finding good sub-populations for modeling

I've built a (deep learning) model for a binary classification problem. For this problem I'm interested in sub-dividing my data set to maximize performance (as measured by precision) by essentially ...
1
vote
0answers
72 views

400 features and 100 classes using weka

I am working on a classification problem, where I have 400 features(all are numeric), and 100 classes and I have 26,000 examples for training. In my project I am using Weka and I have tried different ...
1
vote
0answers
126 views

Bagging for continuous variable

I am using bagging approach in R to predict average Ozone data. I have a confusion that, can I use bagging method to predict average Ozone? Since it is continuous variable. I read some articles where ...
1
vote
0answers
108 views

Class probability with Bagging

What is/are the standard methods to get the class probabilities with Bagging methods? For instance, with Random Forests, one can take the the class proportions at each terminal node. But generally ...
1
vote
0answers
121 views

Reduction of accuracy - Bagging

Bagging is an ensemble method which uses a parallel set of classifiers and then gives consensus output. Usually bagging improves the accuracy of a classification. But if there are situations in which ...
1
vote
0answers
961 views

How to address Boosting and Bagging decreasing the classification accuracy

For my classification, I use several algorithms available in WEKA, but with limited number of features. I got some accuracy levels with the algorithms I used and I tried improving the accuracies using ...
1
vote
0answers
383 views

Can bagging and random subspaces method be used to assemble logistic regression models?

I am after applications and theoretical knowledge of using bagging and random subspaces to assemble logistic regression models, just like how they are used to assemble decision trees to form a random ...
1
vote
0answers
93 views

Which type of bootstrap to use in bagging?

Various bootstrap procedures exist (iid bootstrap, block bootstrap, cross sectional bootstrap...). I was wondering which one I should use when bagging classification trees, which I estimate on panel ...
1
vote
0answers
52 views

Training on time series data with a small number of examples

The data I have is collected from 16 smartphones - it's made up of discrete readings from various sensors (eg. accelerometer in 3 axes, intensity of sound in various frequencies etc.), at regular ...
1
vote
0answers
93 views

Best Random Forest model converging to bagging: What does it mean?

I am performing a grid search to tune the Random Forest parameter $m$ (at $ntree$ fixed). I have $p=79$ variables, and the best model, in terms of $OOB_{error}$, turns out to be a model with $m=76$ ...
1
vote
0answers
590 views

How to merge different predictive models training with different data sets?

Is there any good method to merge/consolidation different predictive models which were trained on different features but outputs the same goal. Example: Model 1 with features a + b + c (trained on ...
1
vote
0answers
72 views

Bagging for sparse survey data regression?

A colleague came to me with an idea for training a regression model from a very large but sparse survey data set. In this data set, many variables are available but most respondents only answered a ...
1
vote
0answers
324 views

How can I estimate (with confidence intervals) the divergence between two smoothing splines?

I have time series data from two independent groups. I want to know whether these groups diverge over time and, if so, when they diverge and for how long. The way I have done this is to estimate (...
0
votes
1answer
17 views

How does bagging effect linear model assumptions?

Linear regression has model assumptions. How does bagging effect model assumptions for linear regression? Also, should you build a bagged linear model with correlated and statistically significant ...
0
votes
0answers
18 views

Stacking of Machine Learning models using different data splits

I have one single dataset with 2 classes. I want to make a model for binary classification, and I am experimenting a bit. My intention is to use stacking on some models by using subsets of the 1 ...
0
votes
2answers
44 views

Is boosting and bagging only relevant in the context of decision trees?

In the documents I've seen on boosting and bagging, it seems that they're always talked about in the context of decision trees. What are some other methods in which the two are applicable?
0
votes
0answers
26 views

Are predictions generated from bootstrap resampling normally distributed?

If I resample a dataset hundreds of times with replacement (bootstrap replications), fit some Model A on each bootstrap sample, and generate predictions (time series) from each fitted model, can I ...
0
votes
0answers
39 views

How are bagging and boosting methods of feature extraction/selection?

I am learning about boosting and bagging from the Amazon Web Services Machine Learning courses. In it, they describe bagging and boosting as ways to automate feature extraction and selection. My ...
0
votes
0answers
212 views

Time complexity of bagging and random forest

Most sources and books state that time complexity of a single decision tree for n points and d dimensions (features) is $O(d * n^2 * log(n))$, with clever caching and one time sorting it’s $O(d * n * ...
0
votes
0answers
6 views

Artificial “Bagging” for GLM poisson

Let say I'm building a model to predict the number of accidents in a insurance portfolio of automobiles. The problem is that my model is is very very sensitive to the seed (for each time I change the ...
0
votes
0answers
14 views

Evaluating baggedETS on a Test set

I'm using the forecast package to test a variety of models on monthly sales for 400 products sold by my company. I'm following the practice of fitting on a test ...
0
votes
0answers
26 views

What is the meaning of component err.rate of class randomForest?

I asked this question on Stackoverflow, but it's likely that I will received no answer on that site. So I cross-post my question here. I'm using the function ...
0
votes
1answer
30 views

Why does creating training sets with replacement lead to better performance?

Pasting generally suffers from lower performance than bagging, because it training sets without replacement. Why does creating training sets with replacement lead to better performance?
0
votes
0answers
35 views

Forecasting methods to work globally on a series of datasets

Forecasting a numeric time-series is providing an estimator $\hat y(t+1)$ of $y(t+1)$ computed from $(y(1),y(2)...y(t))$. You want it to be close to $y(t+1)$ (say in terms of squared distance). Now ...
0
votes
0answers
201 views

In Bagging, how does one train the base learner?

In Bagging (the grouping of predictive variables), I can set a decision tree, a neural network, and so forth as the base learner, since I can get m datasets randomly, and in each dataset, I can train ...
0
votes
0answers
443 views

The Appropriate p (number of predictors)/n (number of observations) Ratio in Tree-based Methods

I am dealing with a dataset in which there are 493 observations spanned over 30 predictors. My intention is to fit a model to make accurate predictions. It seems to me that the ratio $\frac{n}{p}$ ...
0
votes
0answers
326 views

Dynamic Bag of Words / Features

I'm trying to implement a Bag of Features for a set of images submitted in different moments by a set of users. If the clusters change, then we need to recompute at LEAST all the "visual words" which ...