Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

0
votes
1answer
13 views

svm loss function gradient

I was taking Stanford's cs231n class and was unable to understand the gradient calculated using the SVM loss function. You should go here to check the notes which I am talking about. This is the SVM ...
0
votes
1answer
27 views

Backpropagation gradient of the average

In the Pytorch Udacity course, the following is said at one point: To calculate the gradients, you need to run the .backward method on a Variable, ...
1
vote
0answers
30 views

What is state of the art in gradient free neural network learning, esp. for images?

I've been recently looking into gradient free learning of neural networks. However, most of the techniques I've found seem to be only applied to toy problems, which I assume means they're infeasible ...
0
votes
0answers
68 views

Multiclass hinge loss gradient

I am trying to compute the gradient of multi class hinge loss function but i am kinda confused. First things first, I have a W matrix [10xD] (10 classes) that contains the weights. The loss ...
0
votes
1answer
39 views

Interpreting gradient descent as a constrained optimization problem- Reinforcement learning

I' m studying the lectures of Sergey Levine in reinforcement learning, specifically the TRPO algorithm, during his explanation we claims that gradient descent is the same as doing this. He does ...
0
votes
1answer
31 views

Optimisation by using directional derivative

So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise ...
0
votes
0answers
42 views

How is the minimum logarithmic loss calculated when initializing the XGBoost algorithm?

Suppose there are $5$ sample units, $2$ of which carry the feature $y=1$ to be predicted and three of which carry the feature $y=0$. So, $2$ are positive. The XGBoost algorithm initializes with $\...
1
vote
0answers
16 views

gradient vs derivative: defintions of [closed]

According to wikipedia: In mathematics, the gradient is a multi-variable generalization of the derivative. Like the derivative, the gradient represents the slope of the tangent of the graph of ...
4
votes
1answer
81 views

Intuition behind gradient of expected value and logarithm of probabilities

I recently came across the following curious identity: $$\nabla_\theta \mathbb{E}_{x \sim D_\theta}[f(x)] = \mathbb{E}_{x \sim D_\theta} [ \nabla_\theta \log(D_\theta(x)) f(x)],$$ where $D_\theta$ ...
2
votes
0answers
42 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor", they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{...
0
votes
1answer
53 views

Is there an advantage to normalizing labels when using MSE loss?

I am designing a NN that uses MSE as a loss regressor. Its a big network and when I train, the loss/gradients are HUGE. I have to clip my gradients our else the loss just goes to NaNs. The differences ...
2
votes
0answers
30 views

In Gradient Boosting Tree, why do we fit the tree on the residuals and not on the sum of the previous function and the residuals?

In the Gradient Boosting Tree algorithm, as described in https://en.wikipedia.org/wiki/Gradient_boosting#Gradient_tree_boosting, we update the previous model $F_m$ by adding the results $h_m$ of the ...
0
votes
0answers
10 views

Calculating gradient/rate of use

I have some data regarding my pellet usage. I monitor the level of pellets in the hopper, with a certain degree of inaccuracy because of the tesselation of the pellets. I would like to calculate the ...
2
votes
0answers
685 views

Gradient Boosting - Price Forecast based on time series data [closed]

What I am trying to achieve. I want to forecast Natural Gas prices under the column "NG Open" based on other parameters in the data set below for all Contract Months ,which is scraped from a public ...
1
vote
1answer
23 views

Storage and re-computation of Intermediate / Weight / Back-propagated Gradients

I need to track the computation and storage of different parts of my network training. To be on the same page, let's assume the simple following scenario (biases omitted) Questions Local Gradients - ...
0
votes
0answers
24 views

Softmax Gradient Help

I am trying to calculate the gradient of a neural net. Here is the net. Its a 1 layer net with softmax $f_{1}=xW_{1}+b_{2}$ $Y^{*}=S(f_{1})$ $E=-\Sigma_{i}\left(y_{i}log(Y_{i}^{*})\right)$ I am ...
3
votes
0answers
73 views

Gradient Descent in Metric Learning for Kernel Regression (MLKR)

I am currently studying the Metric Learning for Kernel Regression (MLKR) algorithm (http://proceedings.mlr.press/v2/weinberger07a/weinberger07a.pdf). Let $\{(x_{1}, y_{1}), ..., (x_{N}, y_{N})\}$ ...
0
votes
1answer
106 views

focusing on hard examples in neural networks, like in gradient boosting?

gradient boosting can be seen as focusing on the hard examples (the training set examples where the prediction is still far from the true label, and the gradient is still big). is there a similar ...
4
votes
1answer
588 views

Is stochastic gradient descent biased?

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\...
1
vote
0answers
108 views

Get partial derivative in pytorch [closed]

coords[i] is a list containing 3 elements x,y,z and I want to get the derivative of G[i] w.r....
0
votes
1answer
116 views

Gradient in batch-size

When we set a batch-size, after each sample of batch passed we take the gradient but wait until last sample of batch to passed and then propagate the sum of gradient of them through the network? Am I ...
6
votes
1answer
1k views

gradient descent and local maximum

I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
2
votes
1answer
132 views

Determining significant changes in slope in a nested GAM

I have annual measurements from 2 sites and want to plot where significant changes in slope occur in each site (in a nested design). I can achieve this using non-nested data based on Gavin Simpson's ...
8
votes
1answer
827 views

Can I combine many gradient boosting trees using bagging technique

Based on Gradient Boosting Tree vs Random Forest . GBDT and RF using different strategy to tackle bias and variance. My question is that can I resample dataset (with replacement) to train multiple ...
1
vote
1answer
151 views

Find a weak learner in Boosting

I know gradient boosting use an iteration approach to finding a weak learner. But I am confused about the way to find weak learner, PDF source Question 1: Why find the weak learner by the formula ...
0
votes
0answers
1k views

gradient of SVM Loss (hinge loss)

I am trying to follow https://mlxai.github.io/2017/01/06/vectorized-implementation-of-svm-loss-and-gradient-update.html He starts with the SVM loss (hinge loss) function defined per input instance ...
0
votes
0answers
222 views

Gradient function for log-likelihood

We are trying to calculate the gradient for a logistic regression where the log-likelihood function is: $$ll(x,y,\beta)=ln\left(\frac{\Pi {(e^{x_i\beta}})^{y_i}}{\Sigma {e^{x_i\beta}}}\right)$$ $x$ ...
3
votes
1answer
317 views

How do the residual blocks prevent exploding gradients?

I am reading Roger Grosse's lecture notes on ResNet and I have a question regarding the explanation on how residue blocks prevent gradient explosion, see the screenshot below: My confusion is: this ...
1
vote
0answers
195 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
2
votes
0answers
108 views

Generative Adversarial Networks - Gradient saturation

This is the value function from the GANs paper: The authors explain that this equation "may not provide sufficient gradient for $G$ to learn well", because early in the learning process the ...
0
votes
1answer
48 views

Can a full batch gradient descent point not to a minimum for a convex function?

Let's say that we have a three dimentional convex function with a minimum marked by a red dot. Can a full batch gradient in a blue point not directed to a red dot as drawn below? Actually this ...
3
votes
1answer
304 views

Gradient ascent to maximise log likelihood

I'm working on an online method to adapt the parameters $\mu, \Sigma$ of a Gaussian distribution. Do to so i perform a gradient descent on the log likelihood $L$. With the help of the matrix cookbook ...
0
votes
1answer
251 views

Review: Gradient of loss function

This question was asked about the gradient of a linear function. In the answer I don't understand why is $\nabla g(w)=X^t$ (the marked in red part)? Shouldn't it be just $\nabla g(w)=X$? And in ...
2
votes
0answers
274 views

Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
1
vote
1answer
131 views

Gradient in Gradient Boosting

I know the basic overview of how gradient boosting trees work but i am finding it hard to figure out the use of gradient in gradient boosting. My questions may seem stupid but it would be great if ...
1
vote
0answers
66 views

Is REINFORCE equivalent to MLE in degenerate cases?

Consider a simple regression or classification problem using a typical squared loss or cross-entropy loss respectively. The supervised setup uses the gradient as estimated by: $$\nabla_W J \approx \...
1
vote
2answers
510 views

Understanding weak learner splitting criterion in gradient boosting decision tree (lightgbm) paper

I'm trying to understand the description about gradient boosting in the light-gbm paper as in the picture below. (Link to paper: https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-...
0
votes
0answers
312 views

Residuals vs Negative Gradient in Gradient Boosting

In gradient boosting we fit a regression tree to the negative gradient of the loss function. I understand that in case the loss function is the mean square error the two metrics are similar but in ...
3
votes
1answer
206 views

Understanding a proof of conditions for vanishing/exploding gradient in RNNs

I'm looking at some of the preliminaries in understanding vanishing/exploding gradients with recurrent neural networks (RNNs), and I see this paper referenced quite a lot: https://arxiv.org/abs/1211....
2
votes
1answer
1k views

Working for Logistic regression partial derivatives

In Andrew Ng's Neural Networks and Deep Learning course on Coursera the logistic regression loss function for a single training example is given as: $$ \mathcal L(a,y) = - \Big(y\log a + (1 - y)\log (...
1
vote
2answers
168 views

Is computing natural gradient equivalent to deriving directional derivative?

It seems to me that natural gradient is simply derived from directional derivative. For example, for a vector $v$, $\tilde{\nabla} f \cdot v = G^{-1} \nabla f \cdot v = \lim_{h\to0} \frac{f(x+hv)-f(x)...
7
votes
2answers
2k views

Gradient descent on non-convex functions

What situations do we know of where gradient descent can be shown to converge (either to a critical point or to a local/global minima) for non-convex functions? For SGD on non-convex functions, one ...
4
votes
2answers
230 views

Why is two-sided gradient checking more accurate? [closed]

In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking: One-sided difference: $\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - ...
2
votes
1answer
64 views

How backpropagation through gradient descent represents the error after each forward pass

I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how ...
4
votes
1answer
886 views

Gradient and hessian of the MAPE

I want to use MAPE(Mean Absolute Percentage Error) as my loss function. ...
1
vote
1answer
311 views

Huber Loss on top of Cross Entropy

I know that the Huber loss is usually applied on top of the L2 loss in order to prevent exploding gradients. Does it make sense to use the Huber loss on top of the cross entropy loss, though? I have a ...
0
votes
1answer
67 views

Pearlmutter's method for Hessian multiplication

I am trying to understand the abstract below from Pearlmutter's paper. Can someone clarify to me why $R_{\bf{v}}\{\bf{w}\}=\bf{v}$? Thanks a lot!
5
votes
3answers
1k views

what is vanishing gradient?

I have seen the word "vanishing gradient" many times in deep learning literature. what is that? gradient respect to what variable? input variable or hidden units? Does that mean the gradient vector ...
3
votes
1answer
354 views

Calculating t-SNE gradient (a mistake in the original t-SNE paper)

This is specific to the way the gradient of the KL divergence Loss function was derived in the original paper Visualizing Data using tSNE. In the Appendix A (Page 21), where they derive the gradient, ...
3
votes
0answers
165 views

Gradient-informed global optimization

I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following: A hybrid descent method ...