Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

Filter by
Sorted by
Tagged with
35
votes
2answers
13k views

Gradient Boosting for Linear Regression - why does it not work?

While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not ...
12
votes
1answer
8k views

Is gradient boosting appropriate for data with low event rates like 1%?

I am trying gradient boosting on a dataset with event rate about 1% using Enterprise miner, but it is failing to produce any output. My question is, since it a decision tree based approach, is it even ...
9
votes
2answers
4k views

Deriving gradient of a single layer neural network w.r.t its inputs, what is the operator in the chain rule?

Problem is: Derive the gradient with respect to the input layer for a a single hidden layer neural network using sigmoid for input -> hidden, softmax for hidden -> output, with a cross entropy ...
8
votes
3answers
5k views

Numeric Gradient Checking: How close is close enough?

I made a convolutional neural network and I wanted to check that my gradients are being calculated correctly using numeric gradient checking. The question is, how close is close enough? My checking ...
8
votes
3answers
2k views

Gradient descent on non-convex functions

What situations do we know of where gradient descent can be shown to converge (either to a critical point or to a local/global minima) for non-convex functions? For SGD on non-convex functions, one ...
8
votes
1answer
1k views

Can I combine many gradient boosting trees using bagging technique

Based on Gradient Boosting Tree vs Random Forest . GBDT and RF using different strategy to tackle bias and variance. My question is that can I resample dataset (with replacement) to train multiple ...
7
votes
2answers
17k views

How to use XGboost.cv with hyperparameters optimization?

I want to optimize hyperparameters of XGboost using crossvalidation. However, it is not clear how to obtain the model from xgb.cv. For instance I call ...
7
votes
2answers
3k views

Name for outer product of gradient approximation of Hessian

Is there a name for approximating the Hessian as the outer product of the gradient with itself? If one is approximating the Hessian of the log-loss, then the outer product of the gradient with itself ...
6
votes
1answer
2k views

gradient descent and local maximum

I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
6
votes
1answer
5k views

How to compute the gradient and hessian of logarithmic loss? (question is based on a numpy example script from xgboost's github repository)

I would like to understand how the gradient and hessian of the logloss function are computed in an xgboost sample script. I've simplified the function to take numpy arrays, and generated ...
5
votes
3answers
2k views

what is vanishing gradient?

I have seen the word "vanishing gradient" many times in deep learning literature. what is that? gradient respect to what variable? input variable or hidden units? Does that mean the gradient vector ...
5
votes
2answers
630 views

Estimating the gradient of log density given samples

I am interested in estimating the gradient of the log probability distribution $\nabla\log p(x)$ when $p(x)$ is not analytically available but is only accessed via samples $x_i \sim p(x)$. There ...
5
votes
2answers
321 views

Why is two-sided gradient checking more accurate? [closed]

In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking: One-sided difference: $\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - ...
5
votes
1answer
3k views

What is out-of-fold average?

Following is Owen Zhang's slide share. This slide is talking about how to use gradient boosting to deal with data which contains categorical values (cardinality features). I just want to know what ...
5
votes
1answer
938 views

Is stochastic gradient descent biased?

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\...
5
votes
1answer
2k views

Regression with zero inflated continuous response variable using gradient boosting trees and random forest

I have a data set with a lot of 0 values for the continuous response variable (about 50%). I want to understand how well gradient boosting/random forest deals with this problem. My colleague suggested ...
5
votes
0answers
3k views

Bagging of xgboost

The extreme-gradient boosting algorithm seems to be widely applied these days. I often have the feeling that boosted models tend to overfit. I know that there are parameters in the algorithm to ...
4
votes
1answer
1k views

Gradient and hessian of the MAPE

I want to use MAPE(Mean Absolute Percentage Error) as my loss function. ...
4
votes
1answer
2k views

Question with Matrix Derivative: Why do I have to transpose?

In the equation for Recurrent Neural Networks: $$ h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b) $$ Where $h_t$ is of size (N,H) Where $W_{hh}$ is of size (H,H) Where $W_{xh}$ is of size (D,H) Where $...
4
votes
1answer
1k views

Expectation of gradients

I am reading this and am puzzled by equation 8. I don't understand the last bit: why can we move the gradient out of the expectation? $$E_Q[\nabla_\phi\log Q_\phi(h|x)] = E_Q[\frac{\nabla_\phi Q_\phi(...
4
votes
1answer
224 views

Intuition behind gradient of expected value and logarithm of probabilities

I recently came across the following curious identity: $$\nabla_\theta \mathbb{E}_{x \sim D_\theta}[f(x)] = \mathbb{E}_{x \sim D_\theta} [ \nabla_\theta \log(D_\theta(x)) f(x)],$$ where $D_\theta$ ...
4
votes
1answer
920 views

Temporal convolution for NLP [closed]

I'm trying to follow Kalchbrenner et al. 2014 (http://nal.co/papers/Kalchbrenner_DCNN_ACL14) (and basically most of the papers in the last 2 years which applied CNNs to NLP tasks) and implement the ...
3
votes
1answer
2k views

Derive logistic loss gradient in matrix form

User Antoni Parellada had a long derivation here on logistic loss gradient in scalar form. Using the matrix notation, the derivation will be much concise. Can I have a matrix form derivation on ...
3
votes
1answer
10k views

Gradient of loss function for (non)-linear prediction functions

$ \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\xv}{\mathbf{x}} \newcommand{\loss}{L(\wv;\xv, y)} $ I'm trying to clear up the calculation of the gradient of a loss function, ...
3
votes
1answer
31 views

can we use any learners in gradient boosting instead of trees?

As we are simply trying to predict residuals from weak learners and aggregating them, can we use any weak learners in gradient boosting machines instead of trees ? If so, why are the all the gbm ...
3
votes
2answers
3k views

Numerical check of gradient in neural network

I am trying to check if my implementation of backpropogation is correct by checking the calculated gradients with the numeric gradient. I am testing it on a very simple linear network (i.e. no ...
3
votes
2answers
84 views

Why is the expected gradient of a density not parallel to the expected gradient of the log density?

I'm confused by a seemingly counter-intuitive property of the interaction between distributions, log transforms, expectations and gradients. Suppose I have some distribution over random variable $x$ ...
3
votes
1answer
397 views

How do the residual blocks prevent exploding gradients?

I am reading Roger Grosse's lecture notes on ResNet and I have a question regarding the explanation on how residue blocks prevent gradient explosion, see the screenshot below: My confusion is: this ...
3
votes
1answer
398 views

Gradient ascent to maximise log likelihood

I'm working on an online method to adapt the parameters $\mu, \Sigma$ of a Gaussian distribution. Do to so i perform a gradient descent on the log likelihood $L$. With the help of the matrix cookbook ...
3
votes
1answer
244 views

Understanding a proof of conditions for vanishing/exploding gradient in RNNs

I'm looking at some of the preliminaries in understanding vanishing/exploding gradients with recurrent neural networks (RNNs), and I see this paper referenced quite a lot: https://arxiv.org/abs/1211....
3
votes
1answer
402 views

Calculating t-SNE gradient (a mistake in the original t-SNE paper)

This is specific to the way the gradient of the KL divergence Loss function was derived in the original paper Visualizing Data using tSNE. In the Appendix A (Page 21), where they derive the gradient, ...
3
votes
1answer
1k views

Gradient of softmax with cross entropy loss

I'm working on implementing a simple deep model which uses cross-entropy loss, while using softmax to generate predictions. More specifically, I am interested in obtaining the gradient of $$CE(...
3
votes
1answer
942 views

Standardizing numerical and encoding of categorical data for training boosted decision tree

Is there a "best practice" way of standardizing numerical and encoding of categorical data for training boosted decision tree? Both for classification and regression problems
3
votes
0answers
1k views

Gradient Boosting - Price Forecast based on time series data [closed]

What I am trying to achieve. I want to forecast Natural Gas prices under the column "NG Open" based on other parameters in the data set below for all Contract Months ,which is scraped from a public ...
3
votes
0answers
153 views

Gradient Descent in Metric Learning for Kernel Regression (MLKR)

I am currently studying the Metric Learning for Kernel Regression (MLKR) algorithm (http://proceedings.mlr.press/v2/weinberger07a/weinberger07a.pdf). Let $\{(x_{1}, y_{1}), ..., (x_{N}, y_{N})\}$ ...
3
votes
0answers
201 views

Gradient-informed global optimization

I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following: A hybrid descent method ...
3
votes
1answer
433 views

How to compute gradient of partial log-likelihood function in Cox proportional hazards model?

The partial log-likelihood function in Cox proportional hazards is given with such formula $${}_{p}\ell(\beta) = \sum\limits_{i=1}^{K}X_i'\beta - \sum\limits_{i=1}^{K}\log\Big(\sum\limits_{l\in \...
2
votes
3answers
973 views

Is gradient checking useless in high dimensional setting?

Is gradient checking (finite difference for numerical gradient to check if analytical gradient is correct) useless in high dimensional setting (say 100K parameters in a deep neural network)? Here is ...
2
votes
3answers
2k views

Can I checking the correct implementation for gradient descent algorithm by looking at if the loss is monotonically decreasing?

The tricky thing of manually implement optimization algorithm is that, even there are some errors, such as wrong gradient, the algorithm still can work in some way, i.e., decrease the objective, and ...
2
votes
1answer
236 views

Gradient in Gradient Boosting

I know the basic overview of how gradient boosting trees work but i am finding it hard to figure out the use of gradient in gradient boosting. My questions may seem stupid but it would be great if ...
2
votes
2answers
120 views

How to solve MALA when the target density is known up to a constant?

If you look at the wikipedia explanation of Metropolis adjusted Langevin Algorithm, the acceptance ratio is given by The second equation involves taking the gradient of the log of $\pi(x)$. However, ...
2
votes
1answer
1k views

Working for Logistic regression partial derivatives

In Andrew Ng's Neural Networks and Deep Learning course on Coursera the logistic regression loss function for a single training example is given as: $$ \mathcal L(a,y) = - \Big(y\log a + (1 - y)\log (...
2
votes
1answer
835 views

Is the gradient computation in the word2vec implementation actually wrong?

In the paper "Efficient Estimation of Word Representations in Vector Space", it is stated that "All models are trained using stochastic gradient descent and backpropagation": http://arxiv.org/pdf/1301....
2
votes
1answer
181 views

Determining significant changes in slope in a nested GAM

I have annual measurements from 2 sites and want to plot where significant changes in slope occur in each site (in a nested design). I can achieve this using non-nested data based on Gavin Simpson's ...
2
votes
1answer
74 views

How backpropagation through gradient descent represents the error after each forward pass

I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how ...
2
votes
1answer
3k views

XGBoost - Can we find a “better” objective function than RMSE for regression?

If we think back to linear models for a moment, we have Ordinary Least Squares (OLS) versus Generalized Linear Models (GLM). Without going too in-depth, it can be said that GLMs "improve" upon OLS by ...
2
votes
1answer
1k views

Statistically testing for a significant difference between two slope values [closed]

I have five trend lines plotted in excel of number of prescriptions of a 5 different drugs over time (MM/YYYY) and I want to test the statistical significance of the difference between the slopes, to ...
2
votes
2answers
33 views

Simplifying Matrix Form

I'm trying to understand how to obtain the solution to an objective function by solving for theta. I found an example here from Naomi which takes an example from The Elements of Statistical Learning ...
2
votes
1answer
41 views

What is state of the art in gradient free neural network learning, esp. for images?

I've been recently looking into gradient free learning of neural networks. However, most of the techniques I've found seem to be only applied to toy problems, which I assume means they're infeasible ...
2
votes
0answers
51 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor", they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{...