We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

Filter by
Sorted by
Tagged with
5
votes
2answers
319 views

Why is two-sided gradient checking more accurate? [closed]

In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking: One-sided difference: $\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - ...
2
votes
1answer
74 views

How backpropagation through gradient descent represents the error after each forward pass

I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how ...
4
votes
1answer
1k views

Gradient and hessian of the MAPE

I want to use MAPE(Mean Absolute Percentage Error) as my loss function. ...
1
vote
1answer
367 views

Huber Loss on top of Cross Entropy

I know that the Huber loss is usually applied on top of the L2 loss in order to prevent exploding gradients. Does it make sense to use the Huber loss on top of the cross entropy loss, though? I have a ...
0
votes
1answer
75 views

Pearlmutter's method for Hessian multiplication

I am trying to understand the abstract below from Pearlmutter's paper. Can someone clarify to me why $R_{\bf{v}}\{\bf{w}\}=\bf{v}$? Thanks a lot!
5
votes
3answers
2k views

what is vanishing gradient?

I have seen the word "vanishing gradient" many times in deep learning literature. what is that? gradient respect to what variable? input variable or hidden units? Does that mean the gradient vector ...
3
votes
1answer
402 views

Calculating t-SNE gradient (a mistake in the original t-SNE paper)

This is specific to the way the gradient of the KL divergence Loss function was derived in the original paper Visualizing Data using tSNE. In the Appendix A (Page 21), where they derive the gradient, ...
3
votes
0answers
201 views

Gradient-informed global optimization

I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following: A hybrid descent method ...
0
votes
1answer
929 views

Intuition behind Backpropagation gradients

I'm currently taking Andrew Ng's Machine Learning Coursera course, and I'm not sure that I fully understand the Delta gradients in Backpropagation (BP). I see that for the first layer of BP (which ...
-1
votes
2answers
349 views

how do you combine the weak models in gradient boosted tree?

In this article https://www.analyticsvidhya.com/blog/2015/11/quick-introduction-boosting-algorithms-machine-learning/ The author indicates you combine 3 weak models into a final one using gradient ...
2
votes
0answers
1k views

prediction score in decision tree

In this article http://xgboost.readthedocs.io/en/latest/model.html under the caption Tree Ensemble, the diagram shows 3 prediction score under the 3 nodes, +2, +0.1, and -1. How were these numbers ...
1
vote
0answers
362 views

Vanishing gradient problem even after using (leaky) ReLU activations

I have a network that I am training for an image classification task, with about 30 layers. The network is softmax activated at the end and uses cross entropy loss. When I train, I have been noticing ...
1
vote
0answers
30 views

Relationship between sign(weight) and sign(gradient)

On the 4th page of the paper Explaining and Harnessing Adversarial Examples, There is a sentence starts with: Note that the sign of the gradient is just −sign(w), ... But in case of y=-1, for me ...
1
vote
1answer
102 views

Is it necessary to logarithm continuous attributes, when tree construction is histogram powered?

LightGBM wiki says: LightGBM uses the histogram based algorithms, which bucketing continuous feature(attribute) values into discrete bins, to speed up training procedure and reduce memory usage. ...
1
vote
0answers
81 views

How embeddings relates to weights in skip-gram model (word2vec)? [duplicate]

I'm trying to understand how a skip-gram model trains itself. Using the input embedding we predict the probability of the output word. And then using the gradient of the cost function we slightly ...
3
votes
1answer
2k views

Derive logistic loss gradient in matrix form

User Antoni Parellada had a long derivation here on logistic loss gradient in scalar form. Using the matrix notation, the derivation will be much concise. Can I have a matrix form derivation on ...
1
vote
1answer
14k views

how does XGBoost do regression using trees?

Usually my job is to do classification but recently I have a project which requires me to do regression. That is to say, my response variable is not a binary True/False, but a continuous number. the ...
2
votes
0answers
339 views

Deriving the gradient of the loss in SNE

The objective used in SNE is the KL divergence between the two distributions and is given as $$ E(Y) = \sum_i \sum_j p_{j|i}\log \frac{p_{j|i}}{q_{j|i}} $$ and the two distributions are as follows, $...
2
votes
0answers
122 views

Express the density of a function of two random variables using the Gradient and the joint density

I would like to know if it is possible to express the density $f_Z(z)$ of a function $Z = g(X,Y)$ of two continuous "nice" random variables $X$ and $Y$ only using the joint density $f_{XY}(x,y)$ and ...
5
votes
2answers
621 views

Estimating the gradient of log density given samples

I am interested in estimating the gradient of the log probability distribution $\nabla\log p(x)$ when $p(x)$ is not analytically available but is only accessed via samples $x_i \sim p(x)$. There ...
1
vote
0answers
47 views

Subgradient descent: compute subgradient(s) in the point where the objective is nondifferentiable

The task of subgradient descent naturally appears in cases of L1 regularization: $$f(\vec{\theta}) = loss + \lambda\sum{|\theta|}$$ Here we use subderivatives to update current location in a ...
1
vote
0answers
86 views

Scale the gradient of the negative log-likelihood

I am using Discrete-time Unscented Kalman Filter UKF to directly approximate the log-likelihood and its gradient, to be used in the identification of my continuous-discrete dynamic model. It is to be ...
2
votes
0answers
889 views

gradient, with respect to weights, of the cross entropy loss function

I am trying to train a softmax classifier using gradient descent. To do so, I need to be able to calculate the gradient of the loss function, $$ L_i = -\log \left( \frac{e^{f_{y_i}}}{\sum_j e^{f_j}} \...
0
votes
0answers
38 views

Derivative of matrix w.r.t vector [duplicate]

I'm quite out of my element trying to do some matrix calculus. I would like to know what the derivative of $z^{T}y$ w.r.t $z$ is, where z, y are n length vectors. Can anyone suggest good resources ...
0
votes
0answers
307 views

Why can't we backprop the gradients in the recurrent neural nets?

We use back propagation through time in the recurrent neural nets. It's done by adding up the gradients w.r.t to weights in each time step. But my question is why can't we just keep getting the ...
1
vote
0answers
196 views

How to combine gradient noise with optimization methods like Adam

I see two options: Apply gradient noise before you apply an optimization method such as Adam (or just SGD with momentum or something else). I.e. you calculate the gradients, then you add noise to it, ...
1
vote
2answers
135 views

Neural Network, gradients change only in one layer

I have a problem where gradients after minimization using Adam Optimizer only changes from hidden layer to output layer. But not from input layer to hidden layer, it stays the same as the previous ...
0
votes
2answers
1k views

Algorithm to differentiate a Neural Network with respect to features

I have trained a neural network to model a system and I want to use that neural network to optimize a cost function with only a subset of features. My hypothesis function is parameterized by one ...
2
votes
3answers
969 views

Is gradient checking useless in high dimensional setting?

Is gradient checking (finite difference for numerical gradient to check if analytical gradient is correct) useless in high dimensional setting (say 100K parameters in a deep neural network)? Here is ...
2
votes
2answers
120 views

How to solve MALA when the target density is known up to a constant?

If you look at the wikipedia explanation of Metropolis adjusted Langevin Algorithm, the acceptance ratio is given by The second equation involves taking the gradient of the log of $\pi(x)$. However, ...
4
votes
1answer
2k views

Question with Matrix Derivative: Why do I have to transpose?

In the equation for Recurrent Neural Networks: $$ h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b) $$ Where $h_t$ is of size (N,H) Where $W_{hh}$ is of size (H,H) Where $W_{xh}$ is of size (D,H) Where $...
2
votes
3answers
2k views

Can I checking the correct implementation for gradient descent algorithm by looking at if the loss is monotonically decreasing?

The tricky thing of manually implement optimization algorithm is that, even there are some errors, such as wrong gradient, the algorithm still can work in some way, i.e., decrease the objective, and ...
6
votes
1answer
5k views

How to compute the gradient and hessian of logarithmic loss? (question is based on a numpy example script from xgboost's github repository)

I would like to understand how the gradient and hessian of the logloss function are computed in an xgboost sample script. I've simplified the function to take numpy arrays, and generated ...
5
votes
1answer
2k views

Regression with zero inflated continuous response variable using gradient boosting trees and random forest

I have a data set with a lot of 0 values for the continuous response variable (about 50%). I want to understand how well gradient boosting/random forest deals with this problem. My colleague suggested ...
1
vote
0answers
607 views

How to compute the gradients for activation maximization in neural network?

I have a question regarding the Activation Maximization technique for neural networks. Activation Maximization is a technique used to visualize the filters of a neural network: Erhan, Dumitru, et al....
4
votes
1answer
1k views

Expectation of gradients

I am reading this and am puzzled by equation 8. I don't understand the last bit: why can we move the gradient out of the expectation? $$E_Q[\nabla_\phi\log Q_\phi(h|x)] = E_Q[\frac{\nabla_\phi Q_\phi(...
5
votes
1answer
3k views

What is out-of-fold average?

Following is Owen Zhang's slide share. This slide is talking about how to use gradient boosting to deal with data which contains categorical values (cardinality features). I just want to know what ...
0
votes
1answer
2k views

Feature importance in gradient boosted trees

I am tuning the parameters of a gradient boosting regression tree algorithm and find it hard to understand the importance of some variables. Here is the case.. when the number of estimators is ...
2
votes
1answer
3k views

XGBoost - Can we find a “better” objective function than RMSE for regression?

If we think back to linear models for a moment, we have Ordinary Least Squares (OLS) versus Generalized Linear Models (GLM). Without going too in-depth, it can be said that GLMs "improve" upon OLS by ...
1
vote
1answer
3k views

Understanding results of xgboost, parameter tuning [closed]

I ran xgboost with below parameter setting: ...
2
votes
1answer
101 views

constant terms in stochastic gradient descent: when to apply them and how much of the constant gradient component?

in a derivation for the gradient of a collaborative filtering system (similar to Probabilistic Matrix Factorization), I got to the following expression for the gradient of a latent vector $\mathbf{u}...
0
votes
1answer
446 views

Matlab gradient [closed]

In matlab, I must compute the symbolic gradient of a function f(x) with x a vector of dimension 2. For example : ...
0
votes
1answer
37 views

What is the meaning of gradient & progression rate?

Event Rates Over Time for a 60-Year-Old Patient With Baseline Peak Aortic Valve (AV) Gradient of 25 mm Hg and Progression Rate of Aortic Stenosis of 5 mm Hg/Yr I have following questions: Q1. Does ...
1
vote
1answer
5k views

In neural networks, how to compute the mean square error (MSE) in gradient update when using a minibatch?

I've been using a siamese neural network for the binary classification of biological data. I've implemented a Torch version of this algorithm, including a stochastic gradient update function. At ...
2
votes
0answers
308 views

Is there a minimum event rate required for Gradient Boosting to work?

I am trying to run gradient boosting in enterprise miner on a dataset which has event rate of about 2% and sample size is about 1m. It fails to produce any output. Which makes me think, is there a ...
3
votes
1answer
1k views

Gradient of softmax with cross entropy loss

I'm working on implementing a simple deep model which uses cross-entropy loss, while using softmax to generate predictions. More specifically, I am interested in obtaining the gradient of $$CE(...
0
votes
1answer
124 views

Gradient Boosting: Is it possible to use a weak classifier?

My understanding is that a regressor has to be used to fit to the residual. Is it possible to directly apply a classifier? If so, what are the requirements/restrictions?
5
votes
0answers
3k views

Bagging of xgboost

The extreme-gradient boosting algorithm seems to be widely applied these days. I often have the feeling that boosted models tend to overfit. I know that there are parameters in the algorithm to ...
1
vote
0answers
317 views

Is there Early Stop in Stochastic gradient descent?

Let's say I have 100 data points. If stochastic gradient didn't converge at first round of 100 data points, then I need to continue another round. Then if it converged at 50th data point, so total ...
3
votes
1answer
942 views

Standardizing numerical and encoding of categorical data for training boosted decision tree

Is there a "best practice" way of standardizing numerical and encoding of categorical data for training boosted decision tree? Both for classification and regression problems