We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

46 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
3k views

Bagging of xgboost

The extreme-gradient boosting algorithm seems to be widely applied these days. I often have the feeling that boosted models tend to overfit. I know that there are parameters in the algorithm to ...
3
votes
0answers
153 views

Gradient Descent in Metric Learning for Kernel Regression (MLKR)

I am currently studying the Metric Learning for Kernel Regression (MLKR) algorithm (http://proceedings.mlr.press/v2/weinberger07a/weinberger07a.pdf). Let $\{(x_{1}, y_{1}), ..., (x_{N}, y_{N})\}$ ...
3
votes
0answers
201 views

Gradient-informed global optimization

I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following: A hybrid descent method ...
2
votes
1answer
41 views

What is state of the art in gradient free neural network learning, esp. for images?

I've been recently looking into gradient free learning of neural networks. However, most of the techniques I've found seem to be only applied to toy problems, which I assume means they're infeasible ...
2
votes
0answers
51 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor", they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{...
2
votes
0answers
33 views

In Gradient Boosting Tree, why do we fit the tree on the residuals and not on the sum of the previous function and the residuals?

In the Gradient Boosting Tree algorithm, as described in https://en.wikipedia.org/wiki/Gradient_boosting#Gradient_tree_boosting, we update the previous model $F_m$ by adding the results $h_m$ of the ...
2
votes
0answers
119 views

Generative Adversarial Networks - Gradient saturation

This is the value function from the GANs paper: The authors explain that this equation "may not provide sufficient gradient for $G$ to learn well", because early in the learning process the ...
2
votes
0answers
336 views

Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
2
votes
0answers
1k views

prediction score in decision tree

In this article http://xgboost.readthedocs.io/en/latest/model.html under the caption Tree Ensemble, the diagram shows 3 prediction score under the 3 nodes, +2, +0.1, and -1. How were these numbers ...
2
votes
0answers
339 views

Deriving the gradient of the loss in SNE

The objective used in SNE is the KL divergence between the two distributions and is given as $$ E(Y) = \sum_i \sum_j p_{j|i}\log \frac{p_{j|i}}{q_{j|i}} $$ and the two distributions are as follows, $...
2
votes
0answers
122 views

Express the density of a function of two random variables using the Gradient and the joint density

I would like to know if it is possible to express the density $f_Z(z)$ of a function $Z = g(X,Y)$ of two continuous "nice" random variables $X$ and $Y$ only using the joint density $f_{XY}(x,y)$ and ...
2
votes
0answers
891 views

gradient, with respect to weights, of the cross entropy loss function

I am trying to train a softmax classifier using gradient descent. To do so, I need to be able to calculate the gradient of the loss function, $$ L_i = -\log \left( \frac{e^{f_{y_i}}}{\sum_j e^{f_j}} \...
2
votes
1answer
101 views

constant terms in stochastic gradient descent: when to apply them and how much of the constant gradient component?

in a derivation for the gradient of a collaborative filtering system (similar to Probabilistic Matrix Factorization), I got to the following expression for the gradient of a latent vector $\mathbf{u}...
2
votes
0answers
308 views

Is there a minimum event rate required for Gradient Boosting to work?

I am trying to run gradient boosting in enterprise miner on a dataset which has event rate of about 2% and sample size is about 1m. It fails to produce any output. Which makes me think, is there a ...
2
votes
0answers
69 views

Intuition about the gradient of the constraint

I was rethinking the logic of solving constraint optimization problem. And I read many stuff in website like this and textbooks. But the following question is still unsolved (to me). We know that at ...
1
vote
1answer
26 views

Trying to smooth small 'bumps' in graph data using spline interpolation for changepoint detection

I'm trying to detect changes in my data, I want to identify points that are like local minima and shoot upwards. I have used the changepoint package to do so, and upon running it and selecting my ...
1
vote
0answers
52 views

Why do we use the log-derivative trick before Monte Carlo?

I still don't understand how we can approximate the gradient of an expected value... Indeed it's impossible to sample points and then to average the gradients of them as we have only samples... (How ...
1
vote
0answers
27 views

Standard errors for Composite Marginal Likelihood

I am estimating a multivariate ordered probit model using a composite marginal likelihood (CML) approach. In other words, I replace the full likelihood function by a surrogate likelihood constructed ...
1
vote
0answers
218 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
1
vote
0answers
75 views

Is REINFORCE equivalent to MLE in degenerate cases?

Consider a simple regression or classification problem using a typical squared loss or cross-entropy loss respectively. The supervised setup uses the gradient as estimated by: $$\nabla_W J \approx \...
1
vote
0answers
369 views

Vanishing gradient problem even after using (leaky) ReLU activations

I have a network that I am training for an image classification task, with about 30 layers. The network is softmax activated at the end and uses cross entropy loss. When I train, I have been noticing ...
1
vote
0answers
30 views

Relationship between sign(weight) and sign(gradient)

On the 4th page of the paper Explaining and Harnessing Adversarial Examples, There is a sentence starts with: Note that the sign of the gradient is just −sign(w), ... But in case of y=-1, for me ...
1
vote
0answers
47 views

Subgradient descent: compute subgradient(s) in the point where the objective is nondifferentiable

The task of subgradient descent naturally appears in cases of L1 regularization: $$f(\vec{\theta}) = loss + \lambda\sum{|\theta|}$$ Here we use subderivatives to update current location in a ...
1
vote
0answers
87 views

Scale the gradient of the negative log-likelihood

I am using Discrete-time Unscented Kalman Filter UKF to directly approximate the log-likelihood and its gradient, to be used in the identification of my continuous-discrete dynamic model. It is to be ...
1
vote
0answers
196 views

How to combine gradient noise with optimization methods like Adam

I see two options: Apply gradient noise before you apply an optimization method such as Adam (or just SGD with momentum or something else). I.e. you calculate the gradients, then you add noise to it, ...
1
vote
0answers
609 views

How to compute the gradients for activation maximization in neural network?

I have a question regarding the Activation Maximization technique for neural networks. Activation Maximization is a technique used to visualize the filters of a neural network: Erhan, Dumitru, et al....
1
vote
0answers
318 views

Is there Early Stop in Stochastic gradient descent?

Let's say I have 100 data points. If stochastic gradient didn't converge at first round of 100 data points, then I need to continue another round. Then if it converged at 50th data point, so total ...
1
vote
0answers
1k views

Deriving the gradient of a loss function for generalized logistic regression

I am trying, without much success so far, to derive the gradient of the following cost function in order to fit a logistic curve to some data: $J(a, k, b, m) = \sum_i^n(y_i - a + \frac{k - a}{(1 + e^{...
1
vote
0answers
68 views

Gradient of a sum of indicators

EDITED w.r.t. whuber's comment: Say I have a function $\mathbb R^n \rightarrow \mathbb R$: $$f(w_1,\ldots,w_n) = \frac{n^-\sum_{i\in I^-}w_ix_i}{n^+\sum_{i\in I^+}w_ix_i}$$ with fixed $x_i\in\mathbb ...
1
vote
1answer
55 views

RVM gradient of likelihood

Can somebody tell me how Tipping (2001) in his classical paper about the Relevance Vector Machine arrives at the following expressions \begin{align} \mathbf{D} &= (\mathbf{C}^{-1}\mathbf{t}\mathbf{...
0
votes
0answers
14 views

Why doesn't ReLu solve vanishing gradient completely?

As far as I know, ReLu reduces the vanishing gradient problem since its gradient is not very small at larger and smaller values(as long as value is bigger than 0). But my question is this: Why doesn'...
0
votes
0answers
10 views

Whats a good estimation for error measuremets when trying to predict values inside two bands?

I am using gradient boosting to predict two quantiles (upper and lower). The predicted value can be above, below, or in bounds. The problem I am facing is that counting the number of values in bound ...
0
votes
0answers
30 views

Non L2 loss-function in gradient boosting

As I understand the idea of gradient boosting in the (m+1)-th step we take the partial derivatives of the loss with respect to our new parameters $f^{[m]}(x^{(i)})$: $\tilde{y}^{(i)}=-\frac{\partial (\...
0
votes
0answers
13 views

Does increasing the margin value/ delta in a SVM loss function decrease the frequency of coming across kinks when evaluating the gradient?

According to http://cs231n.github.io/optimization-1/, kinks refer to non-differentiable points of a function. Even if the analytical gradient would be zero at such a point, the numerical gradient ...
0
votes
0answers
12 views

Why cache gradients for params between training examples?

I was going through Karpathy's guide here where he defines an simple multiplication gate's forward and backward passes like so ...
0
votes
0answers
85 views

Multiclass hinge loss gradient

I am trying to compute the gradient of multi class hinge loss function but i am kinda confused. First things first, I have a W matrix [10xD] (10 classes) that contains the weights. The loss ...
0
votes
0answers
54 views

How is the minimum logarithmic loss calculated when initializing the XGBoost algorithm?

Suppose there are $5$ sample units, $2$ of which carry the feature $y=1$ to be predicted and three of which carry the feature $y=0$. So, $2$ are positive. The XGBoost algorithm initializes with $\...
0
votes
1answer
100 views

Is there an advantage to normalizing labels when using MSE loss?

I am designing a NN that uses MSE as a loss regressor. Its a big network and when I train, the loss/gradients are HUGE. I have to clip my gradients our else the loss just goes to NaNs. The differences ...
0
votes
0answers
12 views

Calculating gradient/rate of use

I have some data regarding my pellet usage. I monitor the level of pellets in the hopper, with a certain degree of inaccuracy because of the tesselation of the pellets. I would like to calculate the ...
0
votes
0answers
1k views

gradient of SVM Loss (hinge loss)

I am trying to follow https://mlxai.github.io/2017/01/06/vectorized-implementation-of-svm-loss-and-gradient-update.html He starts with the SVM loss (hinge loss) function defined per input instance ...
0
votes
0answers
265 views

Gradient function for log-likelihood

We are trying to calculate the gradient for a logistic regression where the log-likelihood function is: $$ll(x,y,\beta)=ln\left(\frac{\Pi {(e^{x_i\beta}})^{y_i}}{\Sigma {e^{x_i\beta}}}\right)$$ $x$ ...
0
votes
0answers
348 views

Residuals vs Negative Gradient in Gradient Boosting

In gradient boosting we fit a regression tree to the negative gradient of the loss function. I understand that in case the loss function is the mean square error the two metrics are similar but in ...
0
votes
1answer
76 views

Pearlmutter's method for Hessian multiplication

I am trying to understand the abstract below from Pearlmutter's paper. Can someone clarify to me why $R_{\bf{v}}\{\bf{w}\}=\bf{v}$? Thanks a lot!
0
votes
0answers
307 views

Why can't we backprop the gradients in the recurrent neural nets?

We use back propagation through time in the recurrent neural nets. It's done by adding up the gradients w.r.t to weights in each time step. But my question is why can't we just keep getting the ...
0
votes
0answers
749 views

Why does the basic gradient descent not converge for this example?

I have a toy example for a linear regression of the form $$y=\beta_0 + \beta_1x_1 + \beta_2x_2$$ The data is: ...
0
votes
0answers
611 views

Convert time scalar field to velocity vector field

I am trying to figure out if it's possible to create vector velocity field from some raster containing spatial distribution of scalar variable (timestamp of some mass-media event in my case). Here is ...