Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
243

All three are so-called "meta-algorithms": approaches to combine several machine learning techniques into one predictive model in order to decrease the variance (bagging), bias (boosting) or improving the predictive force (stacking alias ensemble). Every algorithm consists of two steps: Producing a distribution of simple ML models on subsets of the ...


137

$\text{error = bias + variance}$ Boosting is based on weak learners (high bias, low variance). In terms of decision trees, weak learners are shallow trees, sometimes even as small as decision stumps (trees with two leaves). Boosting reduces error mainly by reducing bias (and also to some extent variance, by aggregating the output from many models). On the ...


79

Since the interface to xgboost in caret has recently changed, here is a script that provides a fully commented walkthrough of using caret to tune xgboost hyper-parameters. For this, I will be using the training data from the Kaggle competition "Give Me Some Credit". 1. Fitting an xgboost model In this section, we: fit an xgboost model with arbitrary ...


77

Random Forest is a bagging algorithm rather than a boosting algorithm. They are two opposite way to achieve a low error. We know that error can be composited from bias and variance. A too complex model has low bias but large variance, while a too simple model has low variance but large bias, both leading a high error but two different reasons. As a result, ...


65

Bagging: parallel ensemble: each model is built independently aim to decrease variance, not bias suitable for high variance low bias models (complex models) an example of a tree based method is random forest, which develop fully grown trees (note that RF modifies the grown procedure to reduce the correlation between trees) Boosting: sequential ...


56

This is a very interesting question. In order to fully understand what was going on, I had to go through what XGBoost is trying to do, and what other methods we had in our toolbox to deal with it. My answer goes over traditional methods, and how/why XGBoost is an improvement. If you want only the bullet points, there is a summary at the end. Traditional ...


51

This question is addressed in this very nice post. Please take a look at it and the references therein. http://fastml.com/what-is-better-gradient-boosted-trees-or-random-forest/ Notice in the article that the speaks about calibration, and links to another (nice) blog post about it. Still, I find that the paper Obtaining Calibrated Probabilities from ...


49

I'll use the sklearn code, as it is generally much cleaner than the R code. Here's the implementation of the feature_importances property of the GradientBoostingClassifier (I removed some lines of code that get in the way of the conceptual stuff) def feature_importances_(self): total_sum = np.zeros((self.n_features, ), dtype=np.float64) for stage ...


42

Just to elaborate on Yuqian's answer a bit. The idea behind bagging is that when you OVERFIT with a nonparametric regression method (usually regression or classification trees, but can be just about any nonparametric method), you tend to go to the high variance, no (or low) bias part of the bias/variance tradeoff. This is because an overfitting model is ...


41

It is well-known, at least from the late 1960', that if you take several forecasts† and average them, then the resulting aggregate forecast in many cases will outperform the individual forecasts. Bagging, boosting and stacking are all based exactly on this idea. So yes, if your aim is purely prediction then in most cases this is the best you can do. What is ...


34

The caret package can help you optimize the parameter choice for your problem. The caretTrain vignette shows how to tune the gbm parameters using 10-fold repeated cross-validation - other optimization approaches are available it can all run in parallel using the foreach package. Use vignette("caretTrain", package="caret") to read the document. The package ...


33

What am I missing here? I don't think you're really missing anything! Another observation is that a sum of subsequent linear regression models can be represented as a single regression model as well (adding all intercepts and corresponding coefficients) so I cannot imagine how that could ever improve the model. The last observation is that a linear ...


32

I found this introduction may provide some intuitive explanations. In Gradient Boosting, ‘shortcomings’ (of existing weak learners) are identified by gradients. In Adaboost, ‘shortcomings’ are identified by high-weight data points. In my understanding, the exponential loss of Adaboost gives more weights for those samples fitted worse. Anyway, ...


27

My guess is that catboost doesn't use the dummified variables, so the weight given to each (categorical) variable is more balanced compared to the other implementations, so the high-cardinality variables don't have more weight than the others. https://arxiv.org/abs/1706.09516 You want to look at this English language paper from the Yandex team about ...


26

Is overfitting so bad that you should not pick a model that does overfit, even though its test error is smaller? No. But you should have a justification for choosing it. This behavior is not restricted to XGBoost. It is a common thread among all machine learning techniques; finding the right tradeoff between underfitting and overfitting. The formal ...


25

A random forest is not considered a boosting type of algorithm. As explained in your boosting link: ...most boosting algorithms consist of iteratively learning weak classifiers with respect to a distribution and adding them to a final strong classifier. When they are added, they are typically weighted in some way that is usually related to the weak ...


23

Caret package have incorporated xgboost. cv.ctrl <- trainControl(method = "repeatedcv", repeats = 1,number = 3, #summaryFunction = twoClassSummary, classProbs = TRUE, allowParallel=T) xgb.grid <- expand.grid(nrounds = 1000, eta = c(0.01,0.05,0.1), ...


22

See my ensemble learning blog post Sources for this image: Wikipedia sklearn


21

Both of the previous answers are wrong. Package GBM uses interaction.depth parameter as a number of splits it has to perform on a tree (starting from a single node). As each split increases the total number of nodes by 3 and number of terminal nodes by 2 (node $\to$ {left node, right node, NA node}) the total number of nodes in the tree will be $3*N+1$ and ...


19

Suppose you are trying to minimize the objective function via number of iterations. And current value is $100.0$. In given data set, there are no "irreducible errors" and you can minimize the loss to $0.0$ for your training data. Now you have two ways to do it. The first way is "large learning rate" and few iterations. Suppose you can reduce loss by $10.0$ ...


19

No, unless you transform the data. It is a linear model if you transform $x$ using indicator function: $$ x' = \mathbb I \left(\{x>2\}\right) = \begin{cases}\begin{align} 0 \quad &x\leq 2\\ 1 \quad &x>2 \end{align}\end{cases} $$ Then $f(x) = 2x' + 3 = \left(\matrix{3 \\2}\right)^T \left(\matrix{1 \\x'}\right)$ Edit: this was mentioned in the ...


17

I like to think of this in analogy with the case of linear models, and their extension to GLMs (generalized linear models). In a linear model, we fit a linear function to predict our response $$ \hat y = \beta_0 + \beta_1 x_1 + \cdots \beta_n x_n $$ To generalize to other situations, we introduce a link function, which transforms the linear part of the ...


16

As you say a lot has been discussed about this matter, and there's some quite heavy theory that has gone along with it that I have to admit I never fully understood. In my practical experience AdaBoost is quite robust to overfitting, and LPBoost (Linear Programming Boosting) even more so (because the objective function requires a sparse combination of weak ...


16

This may be more in bagging spirit, but nevertheless: If you really have a strong learner, there is no need to improve it by any ensemble stuff. I would say... irrelevant. In blending and bagging trivially, in boosting making a too strong classifier may lead to some breaches in convergence (i.e. a lucky prediction may make the next iteration to predict pure ...


16

What you have displayed is a classic example of overfitting. The small uptick in error comes from poorer performance on the validation portion of your cross-validated data set. More iterations should nearly always improve the error on the training set, but the opposite is true for the validation/test set.


15

So, boosting is a learning algorithm, which can generate high-accuracy predictions using as a subroutine another algorithm, which in turn can efficiently generate hypotheses just slightly better (by an inverse polynomial) than random guessing. It's main advantage is speed. When Schapire presented it in 1990 it was a breakthrough in that it showed that a ...


15

As @aginensky mentioned in the comments thread, it's impossible to get in the author's head, but BRT is most likely simply a clearer description of gbm's modeling process which is, forgive me for stating the obvious, boosted classification and regression trees. And since you've asked about boosting, gradients, and regression trees, here are my plain English ...


14

I talked about this in an answer to a related SO question. Decision trees are just generally a very good fit for boosting, much more so than other algorithms. The bullet point/ summary version is this: Decision trees are non-linear. Boosting with linear models simply doesn't work well. The weak learner needs to be consistently better than random guessing. ...


14

There are two observations needed to understand this implementation. The first is that pred is not a probability, it is a log odds. The second is a standard algebraic manipulation of the binomial deviance that goes like this. Let $P$ be the log odds, what sklearn calls pred. Then the definition of the binomial deviance of an observation is (up to a ...


14

In general, boosting error can increase with the number of iterations, specifically when the data is noisy (e.g. mislabeled cases). This could be your issue, but I wouldn't be able to say without knowing more about your data Basically, boosting can 'focus' on correctly predicting cases that contain misinformation, and in the process, deteriorate the ...


Only top voted, non community-wiki answers of a minimum length are eligible