Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

Filter by
Sorted by
Tagged with
2
votes
1answer
30 views

How Gradient Descent is used for classification with Decision Trees?

I'm not able to see how do we use gradient descent to minimize the loss of binary classification with decision tree. What I understood is that we first have a model (decision tree) that try to predict ...
0
votes
0answers
19 views

Is my understanding and presentation of concept of Gradient Boosting correct?

Initially the model is trained with a training set $\{x_{i}, y_{i}\}_{i=1}^{n}$ by minimizing a differentiable loss function $L(y, F(x))$, and, is initialized with a constant value, \begin{align*} F_{...
1
vote
1answer
45 views

Is a high learning rate irrelevant when dropping the first or last tree in a GBDT with 100 trees?

Suppose we've trained a GBDT model with 100 trees with a fairly high learning rate. Consider two cases: We drop the first tree in the model We drop the last tree in the model We then compare models ...
1
vote
0answers
11 views

How can I obtain group (factor) level Hessian and gradient from GLMM in lme4? [closed]

Summary I am working on a problem where I want to determine group level contributions to the maximum likelihood in a GLMM. To do so, I need to access the Hessian and gradients at the group level. This ...
0
votes
1answer
35 views

Loss function depending on the derivative of a neural network with respect to the input in tensorflow

I have a neural network $x \mapsto f(x, \theta)$, and I can access predictions in my code with out = model(X). Imagine that I have a loss function $l(x,y) = (y-\...
0
votes
1answer
14 views

Initializing gradient boosting with the sample mean

For gradient boosting in the regression setting, the final vector of fitted values is $$F_M(x) = \bar{y} + \rho_1h_1(x) + \ldots, + \rho_M h_M(x)$$ Suppose I have a new data set $x_{new}$ that I'd ...
1
vote
1answer
17 views

Why there is theta in index of gradient symbol in gradient descent update formula for MAML?

In this MAML paper, they use following formula of gradient descent update (see page 3, algorithm 1): $$ \varTheta '\ =\varTheta \ −\ \alpha \nabla _{\varTheta }\mathcal{L}_{\mathcal{T}_{i}}( f_{\...
1
vote
0answers
27 views

Why are we interested in gradient with respect to input?

I am learning about sampling methods for Deep Embedding Learning. I was reading an article named: "Sampling Matters in Deep Embedding Learning" (https://arxiv.org/abs/1706.07567). In the ...
0
votes
0answers
14 views

backword propagation of convolutional neural network layer

Consider a convolutional layer of a convolutional Neural network with a single window applied to a single channel image of size $m\times n$. Considering a window of size $f \times f$ with a stride $s$ ...
0
votes
0answers
27 views

Getting gradient for weights in a softmax classifier

So consider the following scenario: $X$ ($N\times D$) is the input matrix containing all the inputs, $W$ ($D \times C $) is the weight. So $$scores = X.dot(W) $$. Then we're using the softmax ...
1
vote
0answers
18 views

Matrix form of elementwise derivations

The elementwise derivations w.r.t e of $$ J = \frac{1}{2}[\Sigma_{r,s=1}^{R}a_{rt}a_{st}k(e_r,e_s) - 2\Sigma_{r=1}^{R}a_{rt}k(e_r, x_t)]$$ can be given by: $$ \frac{\partial J}{\partial e_r} = \Sigma_{...
0
votes
0answers
15 views

ADAM First and Second Moments

I'm trying to understand why ADAM uses the gradient and the gradient squared respectively as estimators the first and second moments. I was assuming that the random variable we were estimating was the ...
0
votes
0answers
11 views

Gradient check fails for some CNN filters

I've implemented a very basic single layer CNN from scratch in matlab which has a convolutional layer - RELU - linear classifier with soft max and cross entropy loss I used the gradient checking ...
0
votes
1answer
87 views

Question about step size in gradient boosting

Above is the pseudocode for gradient boosting. In Step 2.3, we're computing a multiplier (or step length) $\gamma_m$. Suppose the loss function $L(y_i, \hat{y}_i) = \frac{1}{2}(y_i - \hat{y}_i)^2$. ...
0
votes
0answers
40 views

Gradient of MSE with two sets of parameters

I repropose a question I have had no answer on I am trying to calculate $∇_wMSE=0$ and $∇_mMSE=0$ with '$w$' and '$m$' being matrices of unknown parameters and $MSE=(X⋅m⋅w−Y)^2$ ($X$ and $Y$ are ...
0
votes
0answers
72 views

How to calculate gradient for custom objective function in xgboost for FFORMA

I'm trying to build an implementation of the Feature-based Forecast Model Averaging approach in Python (https://robjhyndman.com/papers/fforma.pdf). However, I'm sort of stuck on computing the gradient ...
0
votes
0answers
15 views

Gradient clipping in high vs low dimensions

Is there a qualitative difference between (l2-norm based) clipping a gradient in a low vs a high dimensional statistical model? In which regime is clipping operation expected to work better?
2
votes
1answer
27 views

A particular method for estimating the gradient of a log-density from samples

Suppose I have $N$ samples $x^1, \ldots, x^N$ which were drawn iid from an unknown density $P(x)$. Suppose I am interested in estimating the vector-valued function $g(x) = \nabla \log P (x)$. One ...
0
votes
1answer
163 views

Lasso Regression

I'm a beginner and hope someone could at least point me a direction on how to write out the gradient for Lasso Regression, thank you so much! $J_{\beta_0,...\beta_m} = \frac{1}{2m}\sum_{i=1}^{m} (y_i ...
1
vote
0answers
28 views

Normalized steepest descent with nuclear/frobenius norm

In steepest gradient descent, we try to find a local minima to a loss function $f(\cdot)$ by the rule: $x_{t} = x - \alpha \triangledown_{x}f(x)$. I've found in textbooks that often we want to ...
0
votes
0answers
46 views

What is the second derivative (Hessian) of normalization function?

The normalization function over a $n$ elements column vector $\mathbf{x} $ is described as: $\frac{\mathbf{x}}{\|\mathbf{x}\|_2}$。(x divide by its L2-norm) The gradient of x is written as : $$\frac{\...
0
votes
0answers
19 views

Why gradient boosting uses Taylor series for the classification problem

The value to determined for the leaf in gradient boosting given by optimising the value of gamma over the loss function. However this is done by using taylor series and dont understand why it is ...
2
votes
0answers
32 views

Why does the dimension of gradient and Hessian matrix not conform for this function?

The function is $f(\mathbf{x}) = e^{-\frac{1}{2}\mathbf{x^TAx}}$, where $\mathbf{A}$ is a square symmetric matrix, and $\mathbf{x}$ is an n-vector. What I found were: $$ \begin{align*} \nabla f ...
2
votes
1answer
210 views

Multiclass gradient boosting: how to derive the initial guess, how to predict a probability

I have some questions regarding multiclass boosted-tree-algorithmus. Currently, I apply xgBoost as implemented in R to solve a multi-classification problem. According to StatQuest, for a simple two-...
0
votes
0answers
7 views

Gradient map based on probability of incident

I have a binary classification model that outputs the probability of an instance belonging to the positive or negative class. I already have the probability threshold by which an instance will be ...
0
votes
0answers
12 views

Autograd theory question

I have attached an image of the mathematical description of calculating the gradient for the cost function from Pytorch. 1.) Is $\vec{y}$ the output of the network? 2.) What is $v$ in terms of a ...
0
votes
0answers
98 views

Valid use of differentiable almost everywhere functions, like hinge loss, in gradient optimization/learners, like SciKit-Learn's SGDClassifier?

So, my abstract question is: is it valid (in the sense that stable convergence is roughly expected) to use functions that are differentiable almost everywhere in the practical application of ...
0
votes
0answers
19 views

Calculate expected gradient length in one-vs-rest or one-vs-one scenario with SVM

According to the paper "Active learning with support vector machines" (LINK) by Kremer et al. from 2014 it is possible to apply the expected gradient length on Support Vector Machines (SVM). Is it ...
3
votes
2answers
78 views

Why don't we use the output of activation functions for explainability instead of Gradient?

Most of the explainability methods use Gradient for measuring the sensitivity of the model to the input features, why can't we use the activation functions themselves as a measure of sensitivity? ...
1
vote
1answer
31 views

Monte Carlo Gradient Estimator [closed]

How do we derive this Monte Carlo Estimator? \begin{equation} \nabla_{\phi}\mathbb{E}_{q_{\phi}(z)}[f(z)] = \mathbb{E}_{q_{\phi}(z)}[f(z) \nabla_{q_{\phi}(z)}\ln q_{\phi}(z)] \end{equation}
0
votes
0answers
64 views

Finding the gradient $\nabla$ of the logistic regression cost function

I want to use vector calculus to derive the gradient $\nabla_wJ(w)$ of the logistic regression cost function $J(w) = -\textbf{y}\cdot ln\textbf{ s} - (\mathbf{1} - \textbf{y}) \cdot ln( \mathbf{1} - \...
0
votes
0answers
51 views

Back-propagation through cross entropy or logistic loss function

I have neural network which ends with softmax function and I want to minimize cross-entropy cost function which takes output of this network and one-hot labels as arguments. To calculate partial ...
0
votes
0answers
16 views

Computing custom gradient for LSTM equations [closed]

Consider an LSTM that takes in as input a sequence of N words $X_1,\cdots,X_N$. Each word is a vector $\in R^D$. The dimension of the LSTM neuron is $H$. Suppose we are doing sentiment classification ...
0
votes
1answer
71 views

Derivation of gradient-bandit algorithm, Why is the sum of the derivatives is zero?

https://www.cs.mcgill.ca/~dprecup/courses/RL/Lectures/2-bandits-2019.pdf In above pdf document, page 19, they explain by formula: $$\sum _{ b }^{ }{ \frac { \nabla { \Pi }_{ t }(b) }{ \nabla { H }_{ ...
0
votes
0answers
106 views

Gradient of multivariate normal distribution function?

Let $X\sim\mathcal{N}_J(\mu,\Sigma)$ be a multivariate normal with PDF $f_X$ and CDF $F_X$. Taking derivatives of $f_X$ wrt $X$, $\mu$ and $\Sigma$ is easy as shown here. However, I am interested in ...
0
votes
0answers
34 views

what's the split criteria used by catboost?

I'm trying to understand the split criteria used by catboost in the "plain" boosting mode (not interested in the "ordered" mode complication). In "algorithm 2 - Building a tree" they are saying that ...
0
votes
1answer
21 views

Computing Gradients for a [-1, 1]-valued RBM

The gradient derivation for a binary-valued RBM with values $\in\{0,1\}$ is well-documented, for example in Goodfellow, et al and here on Cross Validated. However, in some works (e.g., associative ...
0
votes
1answer
94 views

Recurrent Neural Network - Vanishing Gradient in a network that has output at each time step

I am trying to understand the problem of vanishing gradient in RNN. However, it seems to me that this problem is not happen with a network that has output at each time step. Let's say we are trying to ...
0
votes
0answers
26 views

Detecting correct changepoint using cpt.reg and envept

I have used cpt.reg from envcpt and changepoint beta, and I am now getting these results, where I need to detect the blue arrow but it is not. I could change the penalties to try detect it, however ...
0
votes
0answers
39 views

Trying to detect when graph first changes direction

Trying to identify when the point where graph trend changes, the first time it changes, circled in the picture. Any ideas? Prior to the point we want, the graph can be either flat, decreasing or ...
1
vote
1answer
74 views

Trying to smooth small 'bumps' in graph data using spline interpolation for changepoint detection

I'm trying to detect changes in my data, I want to identify points that are like local minima and shoot upwards. I have used the changepoint package to do so, and upon running it and selecting my ...
0
votes
0answers
41 views

Why doesn't ReLu solve vanishing gradient completely?

As far as I know, ReLu reduces the vanishing gradient problem since its gradient is not very small at larger and smaller values(as long as value is bigger than 0). But my question is this: Why doesn'...
1
vote
0answers
139 views

Why do we use the log-derivative trick before Monte Carlo?

I still don't understand how we can approximate the gradient of an expected value... Indeed it's impossible to sample points and then to average the gradients of them as we have only samples... (How ...
0
votes
0answers
11 views

Whats a good estimation for error measuremets when trying to predict values inside two bands?

I am using gradient boosting to predict two quantiles (upper and lower). The predicted value can be above, below, or in bounds. The problem I am facing is that counting the number of values in bound ...
1
vote
0answers
36 views

Standard errors for Composite Marginal Likelihood

I am estimating a multivariate ordered probit model using a composite marginal likelihood (CML) approach. In other words, I replace the full likelihood function by a surrogate likelihood constructed ...
3
votes
2answers
159 views

Why is the expected gradient of a density not parallel to the expected gradient of the log density?

I'm confused by a seemingly counter-intuitive property of the interaction between distributions, log transforms, expectations and gradients. Suppose I have some distribution over random variable $x$ ...
1
vote
1answer
96 views

Non L2 loss-function in gradient boosting

As I understand the idea of gradient boosting in the (m+1)-th step we take the partial derivatives of the loss with respect to our new parameters $f^{[m]}(x^{(i)})$: $\tilde{y}^{(i)}=-\frac{\partial (\...
2
votes
2answers
90 views

Simplifying the Matrix Form of the Solution to Ridge Regression

I'm trying to understand how to obtain the solution to an objective function by solving for the parameter vector $\theta$ in ridge regression. I found an example here from Naomi which takes an example ...
3
votes
1answer
38 views

can we use any learners in gradient boosting instead of trees?

As we are simply trying to predict residuals from weak learners and aggregating them, can we use any weak learners in gradient boosting machines instead of trees ? If so, why are the all the gbm ...
0
votes
1answer
109 views

svm loss function gradient

I was taking Stanford's cs231n class and was unable to understand the gradient calculated using the SVM loss function. You should go here to check the notes which I am talking about. This is the SVM ...