# Tag Info

### How to use XGboost.cv with hyperparameters optimization?

This is how I have trained a xgboost classifier with a 5-fold cross-validation to optimize the F1 score using randomized search for hyperparameter optimization. ...
• 491
Accepted

### How to compute the gradient and hessian of logarithmic loss? (question is based on a numpy example script from xgboost's github repository)

The derivatives are with respect to $x$ (or y_hat in the code) instead of $p$. As you've already derived (Edit: as Simon.H mentioned, since the actual loss should ...
• 16.6k
Accepted

If you do not carefully choose the range of the initial values for the weights, and if you do not control the range of the values of the weights during training, vanishing gradient would occur which ...
• 3,494

### Is gradient boosting appropriate for data with low event rates like 1%?

(To give short answer to this:) It is fine to use a gradient boosting machine algorithm when dealing with an imbalanced dataset. When dealing with a strongly imbalanced dataset it much more relevant ...
• 44.8k
Accepted

### gradient descent and local maximum

If Gradient Descent gets initialized in such a way that it starts at a local maximum (or a saddle point, or a local minimum) with gradient zero, then it will simply stay stuck there. Variations of GD, ...
• 12k

### In GD-optimisation, if the gradient of the error function is w.r.t to the weights, isn't the target value dropped since it's a lone constant?

If we are considering the absolute difference as a norm, that is: $loss(w) = |m_x(w) - t|$ then $\nabla loss(w)$ is far from simply being equivalent to $\nabla m_x(w)$. By definition of the derivative ...
• 679
Accepted

### Gradient and hessian of the MAPE

The Mean Absolute Percentage Error (MAPE) is defined as $$\text{MAPE} := \frac{1}{N}\sum_{i=1}^N\frac{|\hat{y}_i-y_i|}{y_i},$$ where the $y_i$ are actuals and the $\hat{y}_i$ are predictions. The ...
• 126k
Accepted

In short answer, the gradient here refers to the gradient of loss function, and it is the target value for each new tree to predict. Suppose you have a true value $y$ and a predicted value $\hat{y}$. ...
• 2,043
Accepted

### In GD-optimisation, if the gradient of the error function is w.r.t to the weights, isn't the target value dropped since it's a lone constant?

No. A proper Norm will not allow it to be. Even the simplest absolute value function as a loss will depend on $t$: $|m(w)-t|’=\pm m’(w)$, here the sign depends on $t$. TL;DR; Generally, your loss ...
• 61.7k

### Gradient Boosting for Linear Regression - why does it not work?

The least squares projection matrix is given by $X(X^{T}X)^{-1}X^{T}$ We can use this to directly obtain our predicted values $\hat{y}$, e.g. $\hat{y} = X(X^{T}X)^{-1}X^{T}y$ Let's say you fit a ...
• 431
Accepted

### Estimating the gradient of log density given samples

I added a paragraph in 2021, but this is still a 2018 answer. There's been a bunch of recent progress on score-based methods, particularly in diffusion models. This problem is sometimes called score ...
• 25.1k

### Derive logistic loss gradient in matrix form

Here is my try $$J(x) = -\frac{1}{m}\sum_{i = 1}^{m} b_iln(h_i) + (1 - b_i)ln(1 - h_i)$$ where $h_i = \sigma(x^Ta_i)$. Let $A = [a_1^T, \dots, a_m^T]^T$. Assuming $ln, \sigma, \frac{1}{\cdot}$ work ...
• 2,208
Accepted

• 96