Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

Filter by
Sorted by
Tagged with
3
votes
1answer
26 views

can we use any learners in gradient boosting instead of trees?

As we are simply trying to predict residuals from weak learners and aggregating them, can we use any weak learners in gradient boosting machines instead of trees ? If so, why are the all the gbm ...
0
votes
0answers
9 views

Why cache gradients for params between training examples?

I was going through Karpathy's guide here where he defines an simple multiplication gate's forward and backward passes like so ...
0
votes
1answer
25 views

svm loss function gradient

I was taking Stanford's cs231n class and was unable to understand the gradient calculated using the SVM loss function. You should go here to check the notes which I am talking about. This is the SVM ...
0
votes
1answer
28 views

Backpropagation gradient of the average

In the Pytorch Udacity course, the following is said at one point: To calculate the gradients, you need to run the .backward method on a Variable, ...
1
vote
0answers
30 views

What is state of the art in gradient free neural network learning, esp. for images?

I've been recently looking into gradient free learning of neural networks. However, most of the techniques I've found seem to be only applied to toy problems, which I assume means they're infeasible ...
0
votes
0answers
71 views

Multiclass hinge loss gradient

I am trying to compute the gradient of multi class hinge loss function but i am kinda confused. First things first, I have a W matrix [10xD] (10 classes) that contains the weights. The loss ...
0
votes
1answer
40 views

Interpreting gradient descent as a constrained optimization problem- Reinforcement learning

I' m studying the lectures of Sergey Levine in reinforcement learning, specifically the TRPO algorithm, during his explanation we claims that gradient descent is the same as doing this. He does ...
0
votes
1answer
31 views

Optimisation by using directional derivative

So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise ...
0
votes
0answers
49 views

How is the minimum logarithmic loss calculated when initializing the XGBoost algorithm?

Suppose there are $5$ sample units, $2$ of which carry the feature $y=1$ to be predicted and three of which carry the feature $y=0$. So, $2$ are positive. The XGBoost algorithm initializes with $\...
1
vote
0answers
19 views

gradient vs derivative: defintions of [closed]

According to wikipedia: In mathematics, the gradient is a multi-variable generalization of the derivative. Like the derivative, the gradient represents the slope of the tangent of the graph of ...
4
votes
1answer
120 views

Intuition behind gradient of expected value and logarithm of probabilities

I recently came across the following curious identity: $$\nabla_\theta \mathbb{E}_{x \sim D_\theta}[f(x)] = \mathbb{E}_{x \sim D_\theta} [ \nabla_\theta \log(D_\theta(x)) f(x)],$$ where $D_\theta$ ...
2
votes
0answers
46 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor", they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{...
0
votes
1answer
75 views

Is there an advantage to normalizing labels when using MSE loss?

I am designing a NN that uses MSE as a loss regressor. Its a big network and when I train, the loss/gradients are HUGE. I have to clip my gradients our else the loss just goes to NaNs. The differences ...
2
votes
0answers
31 views

In Gradient Boosting Tree, why do we fit the tree on the residuals and not on the sum of the previous function and the residuals?

In the Gradient Boosting Tree algorithm, as described in https://en.wikipedia.org/wiki/Gradient_boosting#Gradient_tree_boosting, we update the previous model $F_m$ by adding the results $h_m$ of the ...
0
votes
0answers
10 views

Calculating gradient/rate of use

I have some data regarding my pellet usage. I monitor the level of pellets in the hopper, with a certain degree of inaccuracy because of the tesselation of the pellets. I would like to calculate the ...
2
votes
0answers
904 views

Gradient Boosting - Price Forecast based on time series data [closed]

What I am trying to achieve. I want to forecast Natural Gas prices under the column "NG Open" based on other parameters in the data set below for all Contract Months ,which is scraped from a public ...
1
vote
1answer
26 views

Storage and re-computation of Intermediate / Weight / Back-propagated Gradients

I need to track the computation and storage of different parts of my network training. To be on the same page, let's assume the simple following scenario (biases omitted) Questions Local Gradients - ...
0
votes
0answers
24 views

Softmax Gradient Help

I am trying to calculate the gradient of a neural net. Here is the net. Its a 1 layer net with softmax $f_{1}=xW_{1}+b_{2}$ $Y^{*}=S(f_{1})$ $E=-\Sigma_{i}\left(y_{i}log(Y_{i}^{*})\right)$ I am ...
3
votes
0answers
91 views

Gradient Descent in Metric Learning for Kernel Regression (MLKR)

I am currently studying the Metric Learning for Kernel Regression (MLKR) algorithm (http://proceedings.mlr.press/v2/weinberger07a/weinberger07a.pdf). Let $\{(x_{1}, y_{1}), ..., (x_{N}, y_{N})\}$ ...
0
votes
1answer
117 views

focusing on hard examples in neural networks, like in gradient boosting?

gradient boosting can be seen as focusing on the hard examples (the training set examples where the prediction is still far from the true label, and the gradient is still big). is there a similar ...
4
votes
1answer
687 views

Is stochastic gradient descent biased?

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\...
1
vote
0answers
118 views

Get partial derivative in pytorch [closed]

coords[i] is a list containing 3 elements x,y,z and I want to get the derivative of G[i] w.r....
0
votes
1answer
139 views

Gradient in batch-size

When we set a batch-size, after each sample of batch passed we take the gradient but wait until last sample of batch to passed and then propagate the sum of gradient of them through the network? Am I ...
6
votes
1answer
1k views

gradient descent and local maximum

I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
2
votes
1answer
150 views

Determining significant changes in slope in a nested GAM

I have annual measurements from 2 sites and want to plot where significant changes in slope occur in each site (in a nested design). I can achieve this using non-nested data based on Gavin Simpson's ...
8
votes
1answer
978 views

Can I combine many gradient boosting trees using bagging technique

Based on Gradient Boosting Tree vs Random Forest . GBDT and RF using different strategy to tackle bias and variance. My question is that can I resample dataset (with replacement) to train multiple ...
1
vote
1answer
160 views

Find a weak learner in Boosting

I know gradient boosting use an iteration approach to finding a weak learner. But I am confused about the way to find weak learner, PDF source Question 1: Why find the weak learner by the formula ...
0
votes
0answers
1k views

gradient of SVM Loss (hinge loss)

I am trying to follow https://mlxai.github.io/2017/01/06/vectorized-implementation-of-svm-loss-and-gradient-update.html He starts with the SVM loss (hinge loss) function defined per input instance ...
0
votes
0answers
230 views

Gradient function for log-likelihood

We are trying to calculate the gradient for a logistic regression where the log-likelihood function is: $$ll(x,y,\beta)=ln\left(\frac{\Pi {(e^{x_i\beta}})^{y_i}}{\Sigma {e^{x_i\beta}}}\right)$$ $x$ ...
3
votes
1answer
344 views

How do the residual blocks prevent exploding gradients?

I am reading Roger Grosse's lecture notes on ResNet and I have a question regarding the explanation on how residue blocks prevent gradient explosion, see the screenshot below: My confusion is: this ...
1
vote
0answers
203 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
2
votes
0answers
110 views

Generative Adversarial Networks - Gradient saturation

This is the value function from the GANs paper: The authors explain that this equation "may not provide sufficient gradient for $G$ to learn well", because early in the learning process the ...
0
votes
1answer
50 views

Can a full batch gradient descent point not to a minimum for a convex function?

Let's say that we have a three dimentional convex function with a minimum marked by a red dot. Can a full batch gradient in a blue point not directed to a red dot as drawn below? Actually this ...
3
votes
1answer
327 views

Gradient ascent to maximise log likelihood

I'm working on an online method to adapt the parameters $\mu, \Sigma$ of a Gaussian distribution. Do to so i perform a gradient descent on the log likelihood $L$. With the help of the matrix cookbook ...
0
votes
1answer
258 views

Review: Gradient of loss function

This question was asked about the gradient of a linear function. In the answer I don't understand why is $\nabla g(w)=X^t$ (the marked in red part)? Shouldn't it be just $\nabla g(w)=X$? And in ...
2
votes
0answers
293 views

Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
1
vote
1answer
154 views

Gradient in Gradient Boosting

I know the basic overview of how gradient boosting trees work but i am finding it hard to figure out the use of gradient in gradient boosting. My questions may seem stupid but it would be great if ...
1
vote
0answers
70 views

Is REINFORCE equivalent to MLE in degenerate cases?

Consider a simple regression or classification problem using a typical squared loss or cross-entropy loss respectively. The supervised setup uses the gradient as estimated by: $$\nabla_W J \approx \...
1
vote
2answers
575 views

Understanding weak learner splitting criterion in gradient boosting decision tree (lightgbm) paper

I'm trying to understand the description about gradient boosting in the light-gbm paper as in the picture below. (Link to paper: https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-...
0
votes
0answers
323 views

Residuals vs Negative Gradient in Gradient Boosting

In gradient boosting we fit a regression tree to the negative gradient of the loss function. I understand that in case the loss function is the mean square error the two metrics are similar but in ...
3
votes
1answer
220 views

Understanding a proof of conditions for vanishing/exploding gradient in RNNs

I'm looking at some of the preliminaries in understanding vanishing/exploding gradients with recurrent neural networks (RNNs), and I see this paper referenced quite a lot: https://arxiv.org/abs/1211....
2
votes
1answer
1k views

Working for Logistic regression partial derivatives

In Andrew Ng's Neural Networks and Deep Learning course on Coursera the logistic regression loss function for a single training example is given as: $$ \mathcal L(a,y) = - \Big(y\log a + (1 - y)\log (...
1
vote
2answers
175 views

Is computing natural gradient equivalent to deriving directional derivative?

It seems to me that natural gradient is simply derived from directional derivative. For example, for a vector $v$, $\tilde{\nabla} f \cdot v = G^{-1} \nabla f \cdot v = \lim_{h\to0} \frac{f(x+hv)-f(x)...
7
votes
2answers
2k views

Gradient descent on non-convex functions

What situations do we know of where gradient descent can be shown to converge (either to a critical point or to a local/global minima) for non-convex functions? For SGD on non-convex functions, one ...
4
votes
2answers
259 views

Why is two-sided gradient checking more accurate? [closed]

In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking: One-sided difference: $\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - ...
2
votes
1answer
69 views

How backpropagation through gradient descent represents the error after each forward pass

I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how ...
4
votes
1answer
997 views

Gradient and hessian of the MAPE

I want to use MAPE(Mean Absolute Percentage Error) as my loss function. ...
1
vote
1answer
327 views

Huber Loss on top of Cross Entropy

I know that the Huber loss is usually applied on top of the L2 loss in order to prevent exploding gradients. Does it make sense to use the Huber loss on top of the cross entropy loss, though? I have a ...
0
votes
1answer
70 views

Pearlmutter's method for Hessian multiplication

I am trying to understand the abstract below from Pearlmutter's paper. Can someone clarify to me why $R_{\bf{v}}\{\bf{w}\}=\bf{v}$? Thanks a lot!
5
votes
3answers
1k views

what is vanishing gradient?

I have seen the word "vanishing gradient" many times in deep learning literature. what is that? gradient respect to what variable? input variable or hidden units? Does that mean the gradient vector ...