Skip to main content
107 votes
Accepted

Explanation of min_child_weight in xgboost algorithm

For a regression, the loss of each point in a node is $\frac{1}{2}(y_i - \hat{y_i})^2$ The second derivative of this expression with respect to $\hat{y_i}$ is $1$. So when you sum the second ...
tkunk's user avatar
  • 1,511
71 votes
Accepted

What is the proper usage of scale_pos_weight in xgboost for imbalanced datasets?

Generally, scale_pos_weight is the ratio of number of negative class to the positive class. Suppose, the dataset has 90 observations of negative class and 10 ...
Harshit Mehta's user avatar
39 votes
Accepted

What algorithms need feature scaling, beside from SVM?

In general, algorithms that exploit distances or similarities (e.g. in the form of scalar product) between data samples, such as k-NN and SVM, are sensitive to feature transformations. Graphical-model ...
yell's user avatar
  • 566
38 votes

What algorithms need feature scaling, beside from SVM?

Here is a list I found on http://www.dataschool.io/comparing-supervised-learning-algorithms/ indicating which classifier needs feature scaling: Full table: In k-means clustering you also need to ...
Franck Dernoncourt's user avatar
38 votes
Accepted

What are the implications of scaling the features to xgboost?

XGBoost is not sensitive to monotonic transformations of its features for the same reason that decision trees and random forests are not: the model only needs to pick "cut points" on features to split ...
Sycorax's user avatar
  • 92.6k
34 votes

Overfitting, but why is the training deviance dropping?

This is exactly what it means to overfit! In many scenarios, you can make the training performance arbitrarily great, perhaps going as far as playing connect-the-dots. This is analogous to your ...
Dave's user avatar
  • 65k
31 votes
Accepted

Mathematical differences between GBM, XGBoost, LightGBM, CatBoost?

...
Alexey Burnakov's user avatar
31 votes
Accepted

What is an intuitive interpretation of the leaf values in XGBoost base learners?

A gradient boosting machine (GBM), like XGBoost, is an ensemble learning technique where the results of the each base-learner are combined to generate the final estimate. That said, when performing a ...
usεr11852's user avatar
30 votes
Accepted

XGBoost vs Python Sklearn gradient boosted trees

You are correct, XGBoost ('eXtreme Gradient Boosting') and sklearn's GradientBoost are fundamentally the same as they are both gradient boosting implementations. However, there are very significant ...
K88's user avatar
  • 444
28 votes

how to avoid overfitting in XGBoost model

XGBoost (and other gradient boosting machine routines too) has a number of parameters that can be tuned to avoid over-fitting. I will mention some of the most obvious ones. For example we can change: ...
usεr11852's user avatar
27 votes
Accepted

How does linear base learner works in boosting? And how does it works in the xgboost library?

Short answer to you question: when the algorithm it fits the residual (or the negative gradient) is it using one feature at each step (i.e. univariate model) or all features (multivariate model)? ...
Haitao Du's user avatar
  • 37.2k
26 votes

How to use XGboost.cv with hyperparameters optimization?

This is how I have trained a xgboost classifier with a 5-fold cross-validation to optimize the F1 score using randomized search for hyperparameter optimization. ...
darXider's user avatar
  • 491
25 votes
Accepted

Gradient for logistic loss function

My answer for my question: yes, it can be shown that gradient for logistic loss is equal to difference between true values and predicted probabilities. Brief explanation was found here. First, ...
Ogurtsov's user avatar
  • 554
25 votes

XGBoost can handle missing data in the forecasting phase

xgboost decides at training time whether missing values go into the right or left node. It chooses which to minimise loss. If there are no missing values at training time, it defaults to sending any ...
Dex Groves's user avatar
  • 1,673
23 votes

Meaning of Surrogate Split

Surrogate splits are referenced elsewhere on this site, but I don't find an explanation for what they are. E.g.: how does rpart handle missing values in predictors? How do decision tree learning ...
Alex's user avatar
  • 4,452
23 votes
Accepted

Is a decision stump a linear model?

No, unless you transform the data. It is a linear model if you transform $x$ using indicator function: $$ x' = \mathbb I \left(\{x>2\}\right) = \begin{cases}\begin{align} 0 \quad &x\leq 2\\ 1 \...
shadowtalker's user avatar
  • 12.8k
23 votes
Accepted

Classification XGBoost vs Logistic Regression

There is no reason for us to expect that a particular type of model $A$ has to be better in terms of performance from another type of model $B$ in every possible use-case. This extends to what is ...
usεr11852's user avatar
22 votes

Overfitting, but why is the training deviance dropping?

Instead of looking at the deviance plot for training and test data we could also take a look at some plots of actual fits. Below is an example of fitting with a polynomial. From left to right the ...
Sextus Empiricus's user avatar
21 votes
Accepted

why boosting method is sensitive to outliers

Outliers can be bad for boosting because boosting builds each tree on previous trees' residuals/errors. Outliers will have much larger residuals than non-outliers, so gradient boosting will focus a ...
Ryan Zotti's user avatar
  • 6,717
21 votes
Accepted

Boosting AND Bagging Trees (XGBoost, LightGBM)

Bagging: Take N random samples of x% of the samples and y% of the Features Instances are repeatedly sub-sampled in Bagging, but not Features. (RandomForests, XGBoost and CatBoost do both): ...
Laksan Nathan's user avatar
20 votes

In boosting, why are the learners "weak"?

I will address overfitting, which hasn't been mentioned yet, with a more intuitive explanation. Your first question was: What are the benefits of using weak as opposed to strong learners? (e.g. why ...
Arthur Colombini Gusmão's user avatar
20 votes

Gradient boosting machine accuracy decreases as number of iterations increases

What you have displayed is a classic example of overfitting. The small uptick in error comes from poorer performance on the validation portion of your cross-validated data set. More iterations should ...
Ryan Zotti's user avatar
  • 6,717
20 votes
Accepted

How to compute the gradient and hessian of logarithmic loss? (question is based on a numpy example script from xgboost's github repository)

The derivatives are with respect to $x$ (or y_hat in the code) instead of $p$. As you've already derived (Edit: as Simon.H mentioned, since the actual loss should ...
dontloo's user avatar
  • 16.6k
19 votes

What is the proper usage of scale_pos_weight in xgboost for imbalanced datasets?

All the documentation says that is should be: scale_pos_weight = count(negative examples)/count(Positive examples) In practice, that works pretty well, but if ...
deltascience's user avatar
19 votes
Accepted

How does gradient boosting calculate probability estimates?

TL;DR: The log-odds for a sample is the sum of the weights of its terminal leafs. The probability of the sample belonging to class 1 is the inverse-logit transformation of the sum. Analogously to ...
Sycorax's user avatar
  • 92.6k
19 votes
Accepted

One hot encoding of a binary feature when using XGBoost

It's true that you're not missing information when you use only $k-1$ categories. In linear models, we are all familiar with the dummy variable trap and the relationship between a model with $k-1$ ...
Sycorax's user avatar
  • 92.6k
18 votes

Intuitive explanations of differences between Gradient Boosting Trees (GBM) & Adaboost

An intuitive explanation of AdaBoost algorithn Let me build upon @Randel's excellent answer with an illustration of the following point In AdaBoost, ‘shortcomings’ are identified by high-weight data ...
Xavier Bourret Sicotte's user avatar
18 votes
Accepted

How xgboost uses weight in the algorithm

It won't be the same. Check this for how XGBoost handles weights: https://github.com/dmlc/xgboost/issues/144 Weighting means increasing the contribution of an example (or a class) to the loss ...
Milad Shahidi's user avatar
17 votes

Poisson deviance (xgboost vs gbm vs regression)

It is not well-documented, but I have examined the source code for xgboost and I have determined the following for the count:poisson objective: It uses the Poisson likelihood with a log link. The <...
Paul's user avatar
  • 11k
17 votes

Calibration curve of XGBoost for binary classification

I'm not sure "the objective function of XGBoost is 'binary:logistic', the probabilities should be well calibrated" is correct: gradient boosting tends to push probability toward 0 and 1. ...
Ben Reiniger's user avatar
  • 4,767

Only top scored, non community-wiki answers of a minimum length are eligible