We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

Filter by
Sorted by
Tagged with
35
votes
2answers
13k views

Gradient Boosting for Linear Regression - why does it not work?

While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not ...
3
votes
1answer
430 views

How to compute gradient of partial log-likelihood function in Cox proportional hazards model?

The partial log-likelihood function in Cox proportional hazards is given with such formula $${}_{p}\ell(\beta) = \sum\limits_{i=1}^{K}X_i'\beta - \sum\limits_{i=1}^{K}\log\Big(\sum\limits_{l\in \...
8
votes
3answers
2k views

Gradient descent on non-convex functions

What situations do we know of where gradient descent can be shown to converge (either to a critical point or to a local/global minima) for non-convex functions? For SGD on non-convex functions, one ...
9
votes
2answers
4k views

Deriving gradient of a single layer neural network w.r.t its inputs, what is the operator in the chain rule?

Problem is: Derive the gradient with respect to the input layer for a a single hidden layer neural network using sigmoid for input -> hidden, softmax for hidden -> output, with a cross entropy ...
5
votes
3answers
2k views

what is vanishing gradient?

I have seen the word "vanishing gradient" many times in deep learning literature. what is that? gradient respect to what variable? input variable or hidden units? Does that mean the gradient vector ...
3
votes
1answer
10k views

Gradient of loss function for (non)-linear prediction functions

$ \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\xv}{\mathbf{x}} \newcommand{\loss}{L(\wv;\xv, y)} $ I'm trying to clear up the calculation of the gradient of a loss function, ...
3
votes
1answer
243 views

Understanding a proof of conditions for vanishing/exploding gradient in RNNs

I'm looking at some of the preliminaries in understanding vanishing/exploding gradients with recurrent neural networks (RNNs), and I see this paper referenced quite a lot: https://arxiv.org/abs/1211....
3
votes
1answer
2k views

Derive logistic loss gradient in matrix form

User Antoni Parellada had a long derivation here on logistic loss gradient in scalar form. Using the matrix notation, the derivation will be much concise. Can I have a matrix form derivation on ...
2
votes
1answer
227 views

Gradient in Gradient Boosting

I know the basic overview of how gradient boosting trees work but i am finding it hard to figure out the use of gradient in gradient boosting. My questions may seem stupid but it would be great if ...
4
votes
1answer
2k views

Question with Matrix Derivative: Why do I have to transpose?

In the equation for Recurrent Neural Networks: $$ h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b) $$ Where $h_t$ is of size (N,H) Where $W_{hh}$ is of size (H,H) Where $W_{xh}$ is of size (D,H) Where $...
3
votes
1answer
942 views

Standardizing numerical and encoding of categorical data for training boosted decision tree

Is there a "best practice" way of standardizing numerical and encoding of categorical data for training boosted decision tree? Both for classification and regression problems
0
votes
1answer
2k views

Feature importance in gradient boosted trees

I am tuning the parameters of a gradient boosting regression tree algorithm and find it hard to understand the importance of some variables. Here is the case.. when the number of estimators is ...