Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

0
votes
0answers
42 views

Multiclass hinge loss gradient

I am trying to compute the gradient of multi class hinge loss function but i am kinda confused. First things first, I have a W matrix [10xD] (10 classes) that contains the weights. The loss ...
0
votes
1answer
39 views

Interpreting gradient descent as a constrained optimization problem- Reinforcement learning

I' m studying the lectures of Sergey Levine in reinforcement learning, specifically the TRPO algorithm, during his explanation we claims that gradient descent is the same as doing this. He does ...
0
votes
1answer
28 views

Optimisation by using directional derivative

So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise ...
0
votes
0answers
37 views

How is the minimum logarithmic loss calculated when initializing the XGBoost algorithm?

Suppose there are $5$ sample units, $2$ of which carry the feature $y=1$ to be predicted and three of which carry the feature $y=0$. So, $2$ are positive. The XGBoost algorithm initializes with $\...
1
vote
0answers
16 views

gradient vs derivative: defintions of [closed]

According to wikipedia: In mathematics, the gradient is a multi-variable generalization of the derivative. Like the derivative, the gradient represents the slope of the tangent of the graph of ...
4
votes
1answer
51 views

Intuition behind gradient of expected value and logarithm of probabilities

I recently came across the following curious identity: $$\nabla_\theta \mathbb{E}_{x \sim D_\theta}[f(x)] = \mathbb{E}_{x \sim D_\theta} [ \nabla_\theta \log(D_\theta(x)) f(x)],$$ where $D_\theta$ ...
2
votes
0answers
37 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor", they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{...
0
votes
1answer
30 views

Is there an advantage to normalizing labels when using MSE loss?

I am designing a NN that uses MSE as a loss regressor. Its a big network and when I train, the loss/gradients are HUGE. I have to clip my gradients our else the loss just goes to NaNs. The differences ...
2
votes
0answers
29 views

In Gradient Boosting Tree, why do we fit the tree on the residuals and not on the sum of the previous function and the residuals?

In the Gradient Boosting Tree algorithm, as described in https://en.wikipedia.org/wiki/Gradient_boosting#Gradient_tree_boosting, we update the previous model $F_m$ by adding the results $h_m$ of the ...
0
votes
0answers
9 views

Calculating gradient/rate of use

I have some data regarding my pellet usage. I monitor the level of pellets in the hopper, with a certain degree of inaccuracy because of the tesselation of the pellets. I would like to calculate the ...
2
votes
0answers
477 views

Gradient Boosting - Price Forecast based on time series data [closed]

What I am trying to achieve. I want to forecast Natural Gas prices under the column "NG Open" based on other parameters in the data set below for all Contract Months ,which is scraped from a public ...
1
vote
1answer
18 views

Storage and re-computation of Intermediate / Weight / Back-propagated Gradients

I need to track the computation and storage of different parts of my network training. To be on the same page, let's assume the simple following scenario (biases omitted) Questions Local Gradients - ...
0
votes
0answers
23 views

Softmax Gradient Help

I am trying to calculate the gradient of a neural net. Here is the net. Its a 1 layer net with softmax $f_{1}=xW_{1}+b_{2}$ $Y^{*}=S(f_{1})$ $E=-\Sigma_{i}\left(y_{i}log(Y_{i}^{*})\right)$ I am ...
3
votes
0answers
60 views

Gradient Descent in Metric Learning for Kernel Regression (MLKR)

I am currently studying the Metric Learning for Kernel Regression (MLKR) algorithm (http://proceedings.mlr.press/v2/weinberger07a/weinberger07a.pdf). Let $\{(x_{1}, y_{1}), ..., (x_{N}, y_{N})\}$ ...
0
votes
1answer
100 views

focusing on hard examples in neural networks, like in gradient boosting?

gradient boosting can be seen as focusing on the hard examples (the training set examples where the prediction is still far from the true label, and the gradient is still big). is there a similar ...
4
votes
1answer
453 views

Is stochastic gradient descent biased?

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\...
1
vote
0answers
89 views

Get partial derivative in pytorch [closed]

coords[i] is a list containing 3 elements x,y,z and I want to get the derivative of G[i] w.r....
0
votes
1answer
71 views

Gradient in batch-size

When we set a batch-size, after each sample of batch passed we take the gradient but wait until last sample of batch to passed and then propagate the sum of gradient of them through the network? Am I ...
6
votes
1answer
877 views

gradient descent and local maximum

I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
2
votes
1answer
106 views

Determining significant changes in slope in a nested GAM

I have annual measurements from 2 sites and want to plot where significant changes in slope occur in each site (in a nested design). I can achieve this using non-nested data based on Gavin Simpson's ...
6
votes
1answer
650 views

Can I combine many gradient boosting trees using bagging technique

Based on Gradient Boosting Tree vs Random Forest . GBDT and RF using different strategy to tackle bias and variance. My question is that can I resample dataset (with replacement) to train multiple ...
1
vote
1answer
132 views

Find a weak learner in Boosting

I know gradient boosting use an iteration approach to finding a weak learner. But I am confused about the way to find weak learner, PDF source Question 1: Why find the weak learner by the formula ...
0
votes
0answers
868 views

gradient of SVM Loss (hinge loss)

I am trying to follow https://mlxai.github.io/2017/01/06/vectorized-implementation-of-svm-loss-and-gradient-update.html He starts with the SVM loss (hinge loss) function defined per input instance ...
0
votes
0answers
207 views

Gradient function for log-likelihood

We are trying to calculate the gradient for a logistic regression where the log-likelihood function is: $$ll(x,y,\beta)=ln\left(\frac{\Pi {(e^{x_i\beta}})^{y_i}}{\Sigma {e^{x_i\beta}}}\right)$$ $x$ ...
3
votes
1answer
284 views

How do the residual blocks prevent exploding gradients?

I am reading Roger Grosse's lecture notes on ResNet and I have a question regarding the explanation on how residue blocks prevent gradient explosion, see the screenshot below: My confusion is: this ...
1
vote
0answers
172 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
2
votes
0answers
103 views

Generative Adversarial Networks - Gradient saturation

This is the value function from the GANs paper: The authors explain that this equation "may not provide sufficient gradient for $G$ to learn well", because early in the learning process the ...
0
votes
1answer
46 views

Can a full batch gradient descent point not to a minimum for a convex function?

Let's say that we have a three dimentional convex function with a minimum marked by a red dot. Can a full batch gradient in a blue point not directed to a red dot as drawn below? Actually this ...
2
votes
1answer
267 views

Gradient ascent to maximise log likelihood

I'm working on an online method to adapt the parameters $\mu, \Sigma$ of a Gaussian distribution. Do to so i perform a gradient descent on the log likelihood $L$. With the help of the matrix cookbook ...
0
votes
1answer
240 views

Review: Gradient of loss function

This question was asked about the gradient of a linear function. In the answer I don't understand why is $\nabla g(w)=X^t$ (the marked in red part)? Shouldn't it be just $\nabla g(w)=X$? And in ...
2
votes
0answers
245 views

Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
1
vote
1answer
103 views

Gradient in Gradient Boosting

I know the basic overview of how gradient boosting trees work but i am finding it hard to figure out the use of gradient in gradient boosting. My questions may seem stupid but it would be great if ...
1
vote
0answers
59 views

Is REINFORCE equivalent to MLE in degenerate cases?

Consider a simple regression or classification problem using a typical squared loss or cross-entropy loss respectively. The supervised setup uses the gradient as estimated by: $$\nabla_W J \approx \...
1
vote
2answers
424 views

Understanding weak learner splitting criterion in gradient boosting decision tree (lightgbm) paper

I'm trying to understand the description about gradient boosting in the light-gbm paper as in the picture below. (Link to paper: https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-...
0
votes
0answers
287 views

Residuals vs Negative Gradient in Gradient Boosting

In gradient boosting we fit a regression tree to the negative gradient of the loss function. I understand that in case the loss function is the mean square error the two metrics are similar but in ...
3
votes
1answer
182 views

Understanding a proof of conditions for vanishing/exploding gradient in RNNs

I'm looking at some of the preliminaries in understanding vanishing/exploding gradients with recurrent neural networks (RNNs), and I see this paper referenced quite a lot: https://arxiv.org/abs/1211....
2
votes
1answer
1k views

Working for Logistic regression partial derivatives

In Andrew Ng's Neural Networks and Deep Learning course on Coursera the logistic regression loss function for a single training example is given as: $$ \mathcal L(a,y) = - \Big(y\log a + (1 - y)\log (...
0
votes
0answers
95 views

Gradient of a Neural Network Estimation of GMM

I am trying to follow this paper, with equations (4), (5), and (6) in which there is a neural network that learns the parameters of a GMM. The paper's objective function wants to minimize E(Z), thus ...
1
vote
2answers
157 views

Is computing natural gradient equivalent to deriving directional derivative?

It seems to me that natural gradient is simply derived from directional derivative. For example, for a vector $v$, $\tilde{\nabla} f \cdot v = G^{-1} \nabla f \cdot v = \lim_{h\to0} \frac{f(x+hv)-f(x)...
7
votes
2answers
1k views

Gradient descent on non-convex functions

What situations do we know of where gradient descent can be shown to converge (either to a critical point or to a local/global minima) for non-convex functions? For SGD on non-convex functions, one ...
4
votes
2answers
185 views

Why is two-sided gradient checking more accurate? [closed]

In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking: One-sided difference: $\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - ...
2
votes
1answer
62 views

How backpropagation through gradient descent represents the error after each forward pass

I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how ...
4
votes
1answer
760 views

Gradient and hessian of the MAPE

I want to use MAPE(Mean Absolute Percentage Error) as my loss function. ...
1
vote
1answer
294 views

Huber Loss on top of Cross Entropy

I know that the Huber loss is usually applied on top of the L2 loss in order to prevent exploding gradients. Does it make sense to use the Huber loss on top of the cross entropy loss, though? I have a ...
0
votes
1answer
63 views

Pearlmutter's method for Hessian multiplication

I am trying to understand the abstract below from Pearlmutter's paper. Can someone clarify to me why $R_{\bf{v}}\{\bf{w}\}=\bf{v}$? Thanks a lot!
5
votes
3answers
1k views

what is vanishing gradient?

I have seen the word "vanishing gradient" many times in deep learning literature. what is that? gradient respect to what variable? input variable or hidden units? Does that mean the gradient vector ...
3
votes
1answer
318 views

Calculating t-SNE gradient (a mistake in the original t-SNE paper)

This is specific to the way the gradient of the KL divergence Loss function was derived in the original paper Visualizing Data using tSNE. In the Appendix A (Page 21), where they derive the gradient, ...
3
votes
0answers
147 views

Gradient-informed global optimization

I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following: A hybrid descent method ...
0
votes
1answer
806 views

Intuition behind Backpropagation gradients

I'm currently taking Andrew Ng's Machine Learning Coursera course, and I'm not sure that I fully understand the Delta gradients in Backpropagation (BP). I see that for the first layer of BP (which ...
-1
votes
2answers
289 views

how do you combine the weak models in gradient boosted tree?

In this article https://www.analyticsvidhya.com/blog/2015/11/quick-introduction-boosting-algorithms-machine-learning/ The author indicates you combine 3 weak models into a final one using gradient ...