Skip to main content
26 votes

How to use XGboost.cv with hyperparameters optimization?

This is how I have trained a xgboost classifier with a 5-fold cross-validation to optimize the F1 score using randomized search for hyperparameter optimization. ...
darXider's user avatar
  • 491
20 votes
Accepted

How to compute the gradient and hessian of logarithmic loss? (question is based on a numpy example script from xgboost's github repository)

The derivatives are with respect to $x$ (or y_hat in the code) instead of $p$. As you've already derived (Edit: as Simon.H mentioned, since the actual loss should ...
dontloo's user avatar
  • 16.6k
17 votes
Accepted

what is vanishing gradient?

If you do not carefully choose the range of the initial values for the weights, and if you do not control the range of the values of the weights during training, vanishing gradient would occur which ...
Hossein's user avatar
  • 3,494
16 votes

Is gradient boosting appropriate for data with low event rates like 1%?

(To give short answer to this:) It is fine to use a gradient boosting machine algorithm when dealing with an imbalanced dataset. When dealing with a strongly imbalanced dataset it much more relevant ...
usεr11852's user avatar
  • 44.8k
15 votes
Accepted

gradient descent and local maximum

If Gradient Descent gets initialized in such a way that it starts at a local maximum (or a saddle point, or a local minimum) with gradient zero, then it will simply stay stuck there. Variations of GD, ...
Skander H.'s user avatar
14 votes

In GD-optimisation, if the gradient of the error function is w.r.t to the weights, isn't the target value dropped since it's a lone constant?

If we are considering the absolute difference as a norm, that is: $loss(w) = |m_x(w) - t|$ then $\nabla loss(w)$ is far from simply being equivalent to $\nabla m_x(w)$. By definition of the derivative ...
mesllo's user avatar
  • 679
12 votes
Accepted

Gradient and hessian of the MAPE

The Mean Absolute Percentage Error (MAPE) is defined as $$\text{MAPE} := \frac{1}{N}\sum_{i=1}^N\frac{|\hat{y}_i-y_i|}{y_i},$$ where the $y_i$ are actuals and the $\hat{y}_i$ are predictions. The ...
Stephan Kolassa's user avatar
11 votes
Accepted

Gradient in Gradient Boosting

In short answer, the gradient here refers to the gradient of loss function, and it is the target value for each new tree to predict. Suppose you have a true value $y$ and a predicted value $\hat{y}$. ...
DiveIntoML's user avatar
  • 2,043
11 votes
Accepted

In GD-optimisation, if the gradient of the error function is w.r.t to the weights, isn't the target value dropped since it's a lone constant?

No. A proper Norm will not allow it to be. Even the simplest absolute value function as a loss will depend on $t$: $|m(w)-t|’=\pm m’(w)$, here the sign depends on $t$. TL;DR; Generally, your loss ...
Aksakal's user avatar
  • 61.7k
10 votes

Gradient Boosting for Linear Regression - why does it not work?

The least squares projection matrix is given by $X(X^{T}X)^{-1}X^{T}$ We can use this to directly obtain our predicted values $\hat{y}$, e.g. $\hat{y} = X(X^{T}X)^{-1}X^{T}y $ Let's say you fit a ...
kirtap's user avatar
  • 431
9 votes
Accepted

Estimating the gradient of log density given samples

I added a paragraph in 2021, but this is still a 2018 answer. There's been a bunch of recent progress on score-based methods, particularly in diffusion models. This problem is sometimes called score ...
Danica's user avatar
  • 25.1k
9 votes

Derive logistic loss gradient in matrix form

Here is my try $$J(x) = -\frac{1}{m}\sum_{i = 1}^{m} b_iln(h_i) + (1 - b_i)ln(1 - h_i)$$ where $h_i = \sigma(x^Ta_i)$. Let $A = [a_1^T, \dots, a_m^T]^T$. Assuming $ln, \sigma, \frac{1}{\cdot}$ work ...
Łukasz Grad's user avatar
  • 2,208
9 votes
Accepted

Understanding weak learner splitting criterion in gradient boosting decision tree (lightgbm) paper

After I obtained some help from the authors, I can write down now how I understand it. Somebody jump in, if there is disagreement. Say, we have some differentiable loss function $L(y,H(x))$ , where $...
kirtap's user avatar
  • 431
8 votes
Accepted

Question with Matrix Derivative: Why do I have to transpose?

As noted in the comments, it is best to write out the matrix equations and then apply the standard derivative rules. After a bit of experience with small cases, where you expand out all the terms, you ...
GeoMatt22's user avatar
  • 13k
8 votes

how does XGBoost do regression using trees?

A regression tree makes sense. You 'classify' your data into one of a finite number of values. Note, that while called a regression, a regression tree is a nonlinear model. Once you believe that, ...
meh's user avatar
  • 2,080
8 votes

what is vanishing gradient?

Consider the following feedforward neural network: Let $w^l_{j,k}$ be the weight for the connection from the $k^{\text{th}}$ neuron in the $(l-1)^{\text{th}}$ layer to the $j^{\text{th}}$ neuron in ...
Oren Milman's user avatar
  • 1,332
8 votes

How Gradient Descent is used for classification with Decision Trees?

Gradient descent is not used for training decision trees. Not every machine learning algorithm uses a general optimization algorithm (e.g. gradient descent) for training, some of them use specialized ...
Tim's user avatar
  • 140k
8 votes
Accepted

What is the meaning of "SGD scales the gradient uniformly in all directions"?

What it's getting at, is the step size you use should depend on the curvature , ie how the gradient changes in each direction. Imagine a narrow u-shaped sloping valley.in the direction of the U, you ...
seanv507's user avatar
  • 7,200
8 votes

Can XGBoost learn more complicated interactions/features?

You are correct. This strongly depends on the amount of data and the nature of the data too. The functional form of $y = a/c + \epsilon$ can actually be quite tricky and in the case of a booster, we ...
usεr11852's user avatar
  • 44.8k
7 votes
Accepted

Can I checking the correct implementation for gradient descent algorithm by looking at if the loss is monotonically decreasing?

There is something "wrong" with the algorithm, gradient descent (its very name is an example of false advertising). That doesn't mean there is a mistake in your implementation. Gradient descent, i.e.,...
Mark L. Stone's user avatar
7 votes
Accepted

Why is two-sided gradient checking more accurate?

One way to look at this is by Taylor approximation. Remember $$f(x+\Delta x)\approx f(x)+\Delta x f'(x)+\frac 1 2 \Delta x^2 f''(x)+\frac 1 6 \Delta x^3f'''(x)+\dots$$ One sided looks like this $$\...
Aksakal's user avatar
  • 61.7k
7 votes

When will gradient descent converge to a critical point or to a local/global minima) for non-convex functions?

In this answer I will explore two interesting and relevant papers that were brought up in the comments. Before doing so, I will attempt to formalize the problem and to shed some light on some of the ...
David Kozak's user avatar
  • 1,842
7 votes
Accepted

Is stochastic gradient descent biased?

For a typical loss function $L = E_{x_i \sim \text{D}}[f(x_i)]$ and true gradient $\nabla L = E[\nabla f(x_i)]$, the expectation of the SGD gradient is $E[\nabla f(x')]$ where $x'$ is the datapoint in ...
shimao's user avatar
  • 26.3k
6 votes
Accepted

What is out-of-fold average?

It's hard to know for sure with such a terse and pithy description, but here's a shot at what he may likely be getting at. Say you have a very high cardinality feature $x$ with some giant set of ...
Matthew Drury's user avatar
6 votes
Accepted

Interpretation of Gradient and Hessian for Categorical variables in Gradient Boost

I understand that the gradient shows the change in the loss function for one unit change in the feature value... Here's the confusion: the gradient (and hessian) is not with respect to the features! ...
Ben Reiniger's user avatar
  • 4,723
6 votes
Accepted

gradient descent in neural network

Neural network models work by superimposing sigmoidal (or other) activation functions to implement the desired mapping. Consider the simple one-dimensional interpolation problem below: There are ...
Dikran Marsupial's user avatar
6 votes

What is the meaning of "SGD scales the gradient uniformly in all directions"?

The gradient vector is multiplied by a scalar constant called learning rate, i.e. $$\theta_{t+1}=\theta_t-\alpha\nabla_\theta L$$ where $\theta$ is the parameter, $L$ is the loss. This means every ...
gunes's user avatar
  • 57.7k
5 votes

Regression with zero inflated continuous response variable using gradient boosting trees and random forest

Updated problem statement Given data: comprised of 50% zero padding, 50% of something else sufficient in size and complexity that xgboost (or equivalent) is needed nonzero data is described by ...
EngrStudent's user avatar
  • 9,500
5 votes
Accepted

Expectation of gradients

$$ E_Q\left[\frac{\nabla_\phi Q_\phi(h|x)}{Q_\phi(h|x)}\right] = \int\frac{\nabla_\phi Q_\phi(h|x)}{Q_\phi(h|x)} Q_\phi(h|x) = \int{\nabla_\phi Q_\phi(h|x)}$$ Assuming that you can exchange the ...
Sale's user avatar
  • 96
5 votes

XGBoost - Can we find a "better" objective function than RMSE for regression?

does the same logic hold true for gradient boosted trees? Yes, by any mean. Gradient boosting can be used to minimize any sensible loss function, and it is very effective in doing it. It is worth ...
carlo's user avatar
  • 4,555

Only top scored, non community-wiki answers of a minimum length are eligible