# Tag Info

Accepted

### Explanation of min_child_weight in xgboost algorithm

For a regression, the loss of each point in a node is $\frac{1}{2}(y_i - \hat{y_i})^2$ The second derivative of this expression with respect to $\hat{y_i}$ is $1$. So when you sum the second ...
• 1,511
Accepted

### What is the proper usage of scale_pos_weight in xgboost for imbalanced datasets?

Generally, scale_pos_weight is the ratio of number of negative class to the positive class. Suppose, the dataset has 90 observations of negative class and 10 ...
• 1,321
Accepted

### What algorithms need feature scaling, beside from SVM?

In general, algorithms that exploit distances or similarities (e.g. in the form of scalar product) between data samples, such as k-NN and SVM, are sensitive to feature transformations. Graphical-model ...
• 566

### What algorithms need feature scaling, beside from SVM?

Here is a list I found on http://www.dataschool.io/comparing-supervised-learning-algorithms/ indicating which classifier needs feature scaling: Full table: In k-means clustering you also need to ...
• 47.2k
Accepted

### What are the implications of scaling the features to xgboost?

XGBoost is not sensitive to monotonic transformations of its features for the same reason that decision trees and random forests are not: the model only needs to pick "cut points" on features to split ...
• 92.6k

### Overfitting, but why is the training deviance dropping?

This is exactly what it means to overfit! In many scenarios, you can make the training performance arbitrarily great, perhaps going as far as playing connect-the-dots. This is analogous to your ...
• 65k
Accepted

...
• 3,119
Accepted

### What is an intuitive interpretation of the leaf values in XGBoost base learners?

A gradient boosting machine (GBM), like XGBoost, is an ensemble learning technique where the results of the each base-learner are combined to generate the final estimate. That said, when performing a ...
• 45k
Accepted

### XGBoost vs Python Sklearn gradient boosted trees

You are correct, XGBoost ('eXtreme Gradient Boosting') and sklearn's GradientBoost are fundamentally the same as they are both gradient boosting implementations. However, there are very significant ...
• 444

### how to avoid overfitting in XGBoost model

XGBoost (and other gradient boosting machine routines too) has a number of parameters that can be tuned to avoid over-fitting. I will mention some of the most obvious ones. For example we can change: ...
• 45k
Accepted

### How does linear base learner works in boosting? And how does it works in the xgboost library?

Short answer to you question: when the algorithm it fits the residual (or the negative gradient) is it using one feature at each step (i.e. univariate model) or all features (multivariate model)? ...
• 37.2k

### How to use XGboost.cv with hyperparameters optimization?

This is how I have trained a xgboost classifier with a 5-fold cross-validation to optimize the F1 score using randomized search for hyperparameter optimization. ...
• 491
Accepted

### Gradient for logistic loss function

My answer for my question: yes, it can be shown that gradient for logistic loss is equal to difference between true values and predicted probabilities. Brief explanation was found here. First, ...
• 554

### XGBoost can handle missing data in the forecasting phase

xgboost decides at training time whether missing values go into the right or left node. It chooses which to minimise loss. If there are no missing values at training time, it defaults to sending any ...
• 1,673

### Meaning of Surrogate Split

Surrogate splits are referenced elsewhere on this site, but I don't find an explanation for what they are. E.g.: how does rpart handle missing values in predictors? How do decision tree learning ...
• 4,452
Accepted

### Is a decision stump a linear model?

No, unless you transform the data. It is a linear model if you transform $x$ using indicator function:  x' = \mathbb I \left(\{x>2\}\right) = \begin{cases}\begin{align} 0 \quad &x\leq 2\\ 1 \...
• 12.8k
Accepted

### Classification XGBoost vs Logistic Regression

There is no reason for us to expect that a particular type of model $A$ has to be better in terms of performance from another type of model $B$ in every possible use-case. This extends to what is ...
• 45k

### Overfitting, but why is the training deviance dropping?

Instead of looking at the deviance plot for training and test data we could also take a look at some plots of actual fits. Below is an example of fitting with a polynomial. From left to right the ...
• 82.2k
Accepted

### why boosting method is sensitive to outliers

Outliers can be bad for boosting because boosting builds each tree on previous trees' residuals/errors. Outliers will have much larger residuals than non-outliers, so gradient boosting will focus a ...
• 6,717
Accepted

### Boosting AND Bagging Trees (XGBoost, LightGBM)

Bagging: Take N random samples of x% of the samples and y% of the Features Instances are repeatedly sub-sampled in Bagging, but not Features. (RandomForests, XGBoost and CatBoost do both): ...
• 2,041

### In boosting, why are the learners "weak"?

I will address overfitting, which hasn't been mentioned yet, with a more intuitive explanation. Your first question was: What are the benefits of using weak as opposed to strong learners? (e.g. why ...

### Gradient boosting machine accuracy decreases as number of iterations increases

What you have displayed is a classic example of overfitting. The small uptick in error comes from poorer performance on the validation portion of your cross-validated data set. More iterations should ...
• 6,717
Accepted

### How to compute the gradient and hessian of logarithmic loss? (question is based on a numpy example script from xgboost's github repository)

The derivatives are with respect to $x$ (or y_hat in the code) instead of $p$. As you've already derived (Edit: as Simon.H mentioned, since the actual loss should ...
• 16.6k

### What is the proper usage of scale_pos_weight in xgboost for imbalanced datasets?

All the documentation says that is should be: scale_pos_weight = count(negative examples)/count(Positive examples) In practice, that works pretty well, but if ...
• 395
Accepted

### How does gradient boosting calculate probability estimates?

TL;DR: The log-odds for a sample is the sum of the weights of its terminal leafs. The probability of the sample belonging to class 1 is the inverse-logit transformation of the sum. Analogously to ...
• 92.6k
Accepted

### One hot encoding of a binary feature when using XGBoost

It's true that you're not missing information when you use only $k-1$ categories. In linear models, we are all familiar with the dummy variable trap and the relationship between a model with $k-1$ ...
• 92.6k

### Intuitive explanations of differences between Gradient Boosting Trees (GBM) & Adaboost

An intuitive explanation of AdaBoost algorithn Let me build upon @Randel's excellent answer with an illustration of the following point In AdaBoost, ‘shortcomings’ are identified by high-weight data ...
Accepted

### How xgboost uses weight in the algorithm

It won't be the same. Check this for how XGBoost handles weights: https://github.com/dmlc/xgboost/issues/144 Weighting means increasing the contribution of an example (or a class) to the loss ...