Hot answers tagged

86

Random Forest is a bagging algorithm rather than a boosting algorithm. They are two opposite way to achieve a low error. We know that error can be composited from bias and variance. A too complex model has low bias but large variance, while a too simple model has low variance but large bias, both leading a high error but two different reasons. As a result, ...


71

Bagging: parallel ensemble: each model is built independently aim to decrease variance, not bias suitable for high variance low bias models (complex models) an example of a tree based method is random forest, which develop fully grown trees (note that RF modifies the grown procedure to reduce the correlation between trees) Boosting: sequential ...


45

Just to elaborate on Yuqian's answer a bit. The idea behind bagging is that when you OVERFIT with a nonparametric regression method (usually regression or classification trees, but can be just about any nonparametric method), you tend to go to the high variance, no (or low) bias part of the bias/variance tradeoff. This is because an overfitting model is ...


41

It is well-known, at least from the late 1960', that if you take several forecasts† and average them, then the resulting aggregate forecast in many cases will outperform the individual forecasts. Bagging, boosting and stacking are all based exactly on this idea. So yes, if your aim is purely prediction then in most cases this is the best you can do. What is ...


27

A random forest is not considered a boosting type of algorithm. As explained in your boosting link: ...most boosting algorithms consist of iteratively learning weak classifiers with respect to a distribution and adding them to a final strong classifier. When they are added, they are typically weighted in some way that is usually related to the weak ...


24

A decision tree works by recursive partition of the training set. Every node $t$ of a decision tree is associated with a set of $n_t$ data points from the training set: You might find the parameter nodesize in some random forests packages, e.g. R: This is the minimum node size, in the example above the minimum node size is 10. This parameter implicitly sets ...


23

No information is passed between trees. In a random forest, all of the trees are identically distributed, because trees are grown using the same randomization strategy for all trees. First, take a bootstrap sample of the data, and then grow the tree using splits from a randomly-chosen subset of features. This happens for each tree individually without ...


22

See my ensemble learning blog post Sources for this image: Wikipedia sklearn


21

The main use-case for bagging is reducing variance of low-biased models by bunching them together. This was studied empirically in the landmark paper "An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants" by Bauer and Kohavi. It usually works as advertised. However, contrary to popular belief, bagging is not ...


13

Bootstrapping is a concept in statistics of approximating the sampling distribution of a statistic by repeatedly sampling from a given sample of size $n$. We construct $B$ samples, each of size $n$, by sampling with replacement from the original sample. The statistic of interest is calculated for each of the $B$ samples. For sufficiently large $B$, we have a ...


13

I do not recommend using an ensemble classifier when your model needs to be interpretable and explainable. Sometimes you need predictions and explanations of the predictions. When you need to convince people that the predictions are worth believing, a highly accurate model can be very persuasive, but I have struggled to convince people to act on ...


13

Bagging: Take N random samples of x% of the samples and y% of the Features Instances are repeatedly sub-sampled in Bagging, but not Features. (RandomForests, XGBoost and CatBoost do both): Given dataset D of size N. For m in n_models: Create new dataset D_i of size N by sampling with replacement from D. Train model on D_i (and then predict) ...


12

In general it is not true that it will always perform better. There are several ensemble methods, each with its own advantages/weaknesses. Which one to use and then depends on the problem at hand. For example, if you have models with high variance (they over-fit your data), then you are likely to benefit from using bagging. If you have biased models, it is ...


12

Parametrical models have parameters (infering them)or assumptions regarding the data distribution, whereas RF ,neural nets or boosting trees have parameters related with the algorithm itself, but they don't need assumptions about your data distribution or classify your data into a theoretical distribution. In fact almost all algorithms have parameters such ...


10

Such questions are always best answered by looking at the code, if you're fluent in Python. RandomForestClassifier.predict, at least in the current version 0.16.1, predicts the class with highest probability estimate, as given by predict_proba. (this line) The documentation for predict_proba says: The predicted class probabilities of an input sample is ...


10

The random forests is a collection of multiple decision trees which are trained independently of one another. So there is no notion of sequentially dependent training (which is the case in boosting algorithms). As a result of this, as mentioned in another answer, it is possible to do parallel training of the trees. You might like to know where the "random" ...


9

Because bagging equalizes influence. This essentially means that the influence of so-called leverage points (points which have a large impact on the overall model) decreases compared to non-bagged models. This is good if the leverage points are bad for the model's performance, which is not always the case. For an example of a leverage point, consider an ...


9

Interesting question. The bootstrap has good sampling properties, compared to some alternatives like the jackknife. The main downside of bootstrapping is that every iteration has to work with a sample that's as big as the original data set (which can be computationally expensive), while some other sampling techniques can work with much smaller samples. ...


9

I would like to answer this question by first overviewing bagging. In bagged trees, we resample observations from a dataset with replacement and fit a tree. We consider all the features in our resampling and this process is repeated $n$ times. If you have ever fit a simple decision tree holding out a test set you will see that your results vary ...


9

By definition, we have $$\operatorname{var}\left(\sum_{i=1}^n{X_i}\right)=\operatorname{cov}\left(\sum_{i=1}^n{X_i},\sum_{i=1}^n{X_i}\right)=\sum_{i=1}^n{\operatorname{var}(X_i)}+\sum_{i\neq j}\operatorname{cov}(X_i,X_j)$$ which is $n \operatorname{var}(X_i)+n(n-1)\operatorname{cov}(X_i,X_j)=n\sigma^2+n(n-1)\rho\sigma^2$, where $i\neq j$. Substituting ...


8

It is an extension of bagging. The procedure is as follows, you take a bootstrap sample of your data and then use this to grow a classification or regression tree (CART). This is done a predefined number of times and the prediction is then the aggregation of the individual trees predictions, it could be a majority vote (for classification) or an average (for ...


8

Random forest is a bagging algorithm rather than a boosting algorithm. Random forest constructs the tree independently using random sample of the data. A parallel implementation is possible. You might like to check out gradient boosting where trees are built sequentially where new tree tries to correct the mistake previously made.


7

Informally, when a model has too high variance it can fit "too well" to the data. That means, that for different data, the parameters of the model found by learning algorithm will be different, or in other words there will be high variance in the learned parameters, depending on the training set. You can think of it that way: data is sampled from some real-...


7

Bagging, boosting, and random forests that have recursive partitioning as the estimator result in a prediction tool that is no longer a tree. That is why these methods have superior predictive accuracy when compared to a single (almost always arbitrary) tree. Recursive partitioning that incorporates cross-validation still results in a tree. In many cases, ...


7

The model that is closest to the true data generating process will always be best and will beat most ensemble methods. So if the data come from a linear process lm() will be much superior to random forests, e.g.: set.seed(1234) p=10 N=1000 #covariates x = matrix(rnorm(N*p),ncol=p) #coefficients: b = round(rnorm(p),2) y = x %*% b + rnorm(N) train=sample(...


7

I am not an expert on the subject, but I think I have a sufficient answer: Since pasting is without replacement, each subset of the sample can be used once at most, which means that you need a big dataset for it to work. As a matter of fact, pasting was originally designed for large data-sets, when computing power is limited. Bagging, on the other hand, can ...


6

Boosting in general has a higher risk of overfitting than bagging. Mislabeled cases cause much more serious trouble with boosting than with bagging, similar to the outliers you mention. I'd expect boosting to yield better results than bagging if you can reasonably expect that your submodels don't by themselves put enough weight on cases close to the class ...


6

In the original paper about bagging, Breiman refers to this point. He explains that unstable learners are likely to give different prediction for modified datasets and likely to benefit from bagging. On the other hand, stable learners (take to the extreme a constant), will give quite similar predictions anyway so bagging won't help. He also refer to ...


6

So how does it works ? Random Forest is a collection of decision trees. The trees are constructed independently. Each tree is trained on subset of features and subset of a sample chosen with replacement. When predicting, say for Classification, the input parameters are given to each tree in the forest and each tree "votes" on the classification, label ...


5

Tal, Generally speaking, pruning will hurt performance of bagged trees. Tress are unstable classifiers; meaning that if you perturb the data a little the tree might significantly change. They are low bias but high variance models. Bagging generally works by "replicating" the model to drive the variance down (the old "increase your sample size" trick). ...


Only top voted, non community-wiki answers of a minimum length are eligible