Skip to main content

Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

Why is the gradient of $S = X \cdot W + b$, $dS$, equal to $\frac{\text{probs} - 1} {N}$, where $\text{probs}$ is the softmax classifier of $S$?

Why is the gradient of $S = X \cdot W + b$, $dS$, equal to $\frac{\text{probs} - 1} {N}$? Here, probs is a $1 \times K$ array which represents the probabilities that $X_i$ belongs to class $k$, where $...
Yash Jain's user avatar
  • 101
0 votes
0 answers
19 views

Understanding softmax as an activation function, and sparsity in data and gradients

I’m working on a project that includes a probabilistic model that uses one hots, and also occasionally partially freezes weights or zeros gradients to specific regions of the weights. In some parts of ...
Danny's user avatar
  • 1
0 votes
0 answers
43 views

Derivative of the multivariate normal cumulative distribution function (CDF) with reparameterisation [duplicate]

I would like to learn how to calculate the derivatives of a multivariate normal cumulative distribution function (MVN CDF) w.r.t. certain elements by using the derivatives of the same MVN CDF w.r.t. ...
Kirin G.'s user avatar
0 votes
0 answers
10 views

Gradient flow through sampled tokens when training RNNs (but without teacher forcing)

Suppose we want to train an autoregressive generative language model based on a recurrent neural network (RNN) architecture without teacher forcing: At each timestep, the RNN takes an input token $x_t$...
Ben JW's user avatar
  • 101
0 votes
0 answers
47 views

BPTT in multi-layer RNN

I am trying to understand the Backpropagation through time algorithm for multi-layer RNNs but I'm facing challenges in extending the process from single-layer to multi-layer architectures. I have ...
Alison's user avatar
  • 1
0 votes
0 answers
14 views

Understanding of the FGSM attack by Goodfellow et al [duplicate]

The authors of the paper "Explaining and harnessing adversarial examples" (ICLR'15) give the following rationale for taking the sign of the gradient of the loss with respect to its inputs in ...
synack's user avatar
  • 371
0 votes
0 answers
38 views

When the expected value of the gradient of function is equal to the expected value of the function multiplied by the gradient itself?

I'm having doubts on the conditions of an equality. Considering $f(X,w): \mathbb{R}^n \xrightarrow{} \mathbb{R}^n$ a function of n-variate random variable $X$ with an unknown distribution $p(x)$ and $...
GM_'s user avatar
  • 1
0 votes
0 answers
47 views

What are the benefits of using pseudo-residuals in Gradient Boosting?

At each iteration $t$ of the Gradient Boosting algorithm, we're basically trying to add the weak learner $f_t$ that minimizes: $$ \mathcal{L}_t = \sum\limits_{i=1}^{n} l(y_i, \hat{y}_i^{(t-1)} + f_t(\...
Druudik's user avatar
  • 143
0 votes
0 answers
29 views

Manual Gradient Computation and Weight Update in PyTorch

I do not want to use torch’s default loss.backward function for gradient computation. Instead I am calculating the gradients manually from the loss function (via torch.autograd.grad). But my gradients ...
rajoy99's user avatar
  • 21
0 votes
0 answers
35 views

How does the chain-rule look for the gradient of a loss function?

When we are computing the gradient of the loss function, $L$, of a Word2Vec model, for the context word-embedding, $w_i$, and the target word-embedding, $t$. Where the loss function, $L$, looks like: $...
ZenPyro's user avatar
5 votes
1 answer
94 views

Get unbiased derivative from GAM

I have time-dependent data and I'm interested in the gradient at time zero. From theory I expect the gradient of the true relationship to be highest at time zero. I would like to estimate the ...
Roland's user avatar
  • 6,701
1 vote
0 answers
10 views

Issues with gradient of standard deviation in GPR using skopt.learning.gaussian_process

I'm currently working on a Gaussian Process Regression (GPR) model using the implementation provided in skopt.learning.GaussianProcessRegressor which is a wrapper for the sklearn implementation. This ...
Dave's user avatar
  • 11
0 votes
0 answers
28 views

Partial Differential Equations for Image Inpainting - Gradients and Orthogonal to gradients

I hope this is the correct channel. Following the Coursera course on image inpainting using PDEs [1], the instructor presents a method for image Inpainting that follows the next equation: $\frac{\...
mgbacher's user avatar
  • 121
2 votes
0 answers
47 views

What algorithm can find a gradient increase using the maximum value of small circles on a two dimensional plot?

I'm looking to make research on an algorithm that finds the maximum increase (gradient) on a 2D-landscape using the maximum values found on circles. The algorithm looks at the values on each circle to ...
M. Beausoleil's user avatar
1 vote
0 answers
46 views

Metropolis Hastings Proposal with Gradient and Hessian Information

I need to sample a high-dimensional parameter vector from a distribution where the gradient, the Hessian and the inverse of the Hessian of the log-likelihood are very cheap to compute. Are there any ...
yrx1702's user avatar
  • 700
0 votes
0 answers
37 views

XGBoost's subsample = 0?

In my use of XGBoost with the gradient-based method, I inadvertently set subsample to 0, yet it surprisingly returned a good result. I am not sure how to explain it well. Any idea from the community?
Mel Huang's user avatar
4 votes
1 answer
126 views

Can XGBoost learn more complicated interactions/features?

For a set of features {a, b, c, d . . . n}, XGBoost can easily learn, say, a*d. In practice can it also effectively learn a/c? Or (a + b + c + 2)/d? Or (c^(2d))/(b^a)? I'd imagine some of this depends ...
BigMistake's user avatar
1 vote
0 answers
62 views

Gradient of Gaussian Process Regressor

I have a data ((x,y),f) that I am fitting using Gaussian Process Regression in Python's sklearn package. The posterior mean of the GP is essentially my output with an associated error. Based on either ...
Prince SBI's user avatar
5 votes
2 answers
429 views

What is the meaning of "SGD scales the gradient uniformly in all directions"?

I'm really newbie about neural network and optimization. When I read the references, I found this journal Wang et al 2018. The journal stated: One disadvantage of SGD is that it scales the gradient ...
andryan86's user avatar
  • 127
2 votes
1 answer
71 views

Exploding/Vanishing gradients deeper understanding

I'm trying to gain deper understanding of the logic behind vanishing and exploding gradients. Most sources I've come across explain the problem by saying that when the weights become too small, the ...
Cipollino's user avatar
1 vote
0 answers
72 views

Closed form expression for the gradient of a fully connected neural network with respect to its parameters

A real-valued feedforward/fully connected neural network with activation function $\sigma : x\in \mathbb R \mapsto \max \{0,x\}\equiv \text{ReLU}(x)$ can formally be seen as a function $f_\theta :\...
Stratos supports the strike's user avatar
0 votes
0 answers
73 views

How do you find the gradients of weights and biases in neural network during back propagation?

I have been trying to create a neural network from scratch. I have been trying to calculate the gradients of the weights and biases of the neural network by watching videos and reading papers, but ...
ManOnTheMoon's user avatar
0 votes
0 answers
66 views

How to measure the noise in gradients of deep neural networks

I'm dealing with a problem related to explainable AI. The short version of the issue I have is that I've observed some interesting differences between large and small BERT models and I suspect that ...
Aitak Aitov's user avatar
1 vote
0 answers
67 views

Explainable AI - Noise in gradients and embeddings of large language models

I am doing experiments related to explainable AI. I have two BERT models - the standard bert-base-cased and a distilled ...
Aitak Aitov's user avatar
1 vote
0 answers
50 views

Neural network parameters dependency vs gradient descent [duplicate]

Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we ...
Glue's user avatar
  • 485
1 vote
0 answers
38 views

Which training instances are being subsampled in each iteration of XGBoost?

The subsample option in XGBoost is described here as follows: Subsample ratio of the training instances. Setting it to 0.5 ...
Anders's user avatar
  • 11
2 votes
0 answers
89 views

Accelerate the fitting of an ECM-GARCH model by computing MLE gradient numerically?

I'm trying to fit an ECM model with variance following a GARCH-DCC model (GARCH with dynamic cross correlation). It has 16 parameters for 2 assets (ECM : 4 gammas, 2 lambda, GARCH: 2 alphas, 2 beta, 2 ...
Jerem Lachkar's user avatar
1 vote
0 answers
48 views

Doubts about Gradient and Matrices derivatives

Studying NN and I wanted to grasp from scratch the theory of gradient descent and matrices derivatives, so I took a simple scenario and tried to apply gradient ascent and see if everything made sense. ...
paolopazzo's user avatar
2 votes
0 answers
100 views

Why doesn't Logloss match Similarity score in this xgboost example?

I am trying to get direct connection between Gain and Logloss for XGBoost. It looks to me that in Xgboost paper formula 6: for a model with depth 1 and number of trees=1 this formula contains ...
FedorT54's user avatar
1 vote
1 answer
248 views

Gain vs Loss in terms of selecting best leaf split value

In the XGBoost Documentation they specify the Gain term as: \begin{equation} Gain=\frac{1}{2} \left[ \frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda}- \frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\...
FedorT54's user avatar
3 votes
1 answer
128 views

Inference of Gradient Boosting on test istance

According to my professor, Gradient Boosting can be done using the following algorithm: Now, I do not really understand the inference part of that algorithm. Why cannot we not simply return $F^{(K)}(...
kklaw's user avatar
  • 525
6 votes
0 answers
730 views

Why do we minimise a cost function instead of maximising an equivalent? [duplicate]

I don't really understand why we minimise a cost function for gradient descent. Why don't we try to have something like a gradient 'climb', where we maximise some function? Is it due to convention, or ...
Y-MinG's user avatar
  • 61
0 votes
0 answers
31 views

Fast renormalization of distributions

Say I have a matrix probs $\in N\times4$ where each row is a categorical distribution. However I have also a mask $\in N\times4$ that is meant to remove actions that are not available. What I used to ...
Alberto's user avatar
  • 1,207
4 votes
2 answers
372 views

Why there is high variance of gradients estimated in the short directions in regression?

I was trying to understand Ridge Regression and came across the following excerpt from Hastie et al. in The Elements of Statistical Learning (section 3.4.1, Page 67): If we consider fitting a linear ...
Rohan Prasad's user avatar
4 votes
1 answer
574 views

Gradient and Hessian of loss function

I'm trying to clear up the calculation of the gradient and Hessian of a loss function in an article that I am currently reading. The loss function is given by $$\ell(\beta)=\sum_{i=1}^{N} e^{-y_{i}{{x}...
ADAM's user avatar
  • 731
1 vote
0 answers
45 views

Is there anyway to calculate the integral of a trace? [closed]

I would like to calculate the integral of a scalar function as follows: $$f(x)=\mathrm{tr}((\mathbf{A}x+\mathbf{B})^{-1}\mathbf{B}),$$ where $\mathbf{A}$ and $\mathbf{B}$ are two $n\times n$ positive-...
Deku's user avatar
  • 21
1 vote
0 answers
37 views

How to merge 2 losses in a reasonable ratio

my question is pretty basic, but I can't find many resources online about this. Say we have two losses for a model, for example a pix2pix GAN, which for those who are not familiar with it, for the ...
Alberto's user avatar
  • 1,207
1 vote
1 answer
91 views

Making sense of the Gain term in Gradient tree boosting

In the XGBoost Documentation they specify the Gain term as \begin{equation} Gain=\frac{1}{2} \left[ \frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda}- \frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\...
Hadar's user avatar
  • 125
2 votes
1 answer
269 views

Why do we regularize large gradients corresponding to large errors?

While reviewing some scientific blogs, I found them recommending using gradient clipping for large error gradients. However, intuitively one would think that when model predictions are completely off, ...
desert_ranger's user avatar
1 vote
1 answer
112 views

Lowering the weight of particular features in a neural network?

Given sample data $x$, we hypothesize that some features (i.e. dimensions) of $x$ will generalize well, while others will generalize poorly. For example, when predicting medical diagnosis, age and ...
SRobertJames's user avatar
0 votes
1 answer
376 views

Need for reparameterization trick in RL (and others)?

This is a multi-fold question that has a number of closely related questions; that is why I will pose them all here, instead of separate questions. In RL you have a parameterized policy that dictates ...
Schach21's user avatar
  • 145
1 vote
0 answers
9 views

Is backprop computed for every element in the minibatch and then averaged for every weight?

I'm trying to fully understand backpropagation by computing it by hand. Often is cited that is just an alternation of the derivative of the preactivation and the derivative of the activation, however, ...
Alberto's user avatar
  • 1,207
4 votes
1 answer
1k views

Can Poisson deviance be used to evaluate models that use loss functions other than Poisson? (Such as MSE)

I am currently doing a a study on emergency department utilization rates at various geography levels. Especially of interest, are tree-based approaches to this analysis - namely random forest and GBMs....
jmc8's user avatar
  • 143
1 vote
0 answers
230 views

How does the full derivative of softmax + cross entropy have the correct dimensions?

The blog post the softmax function and its derivative explains the following: Imagine that each input has $N$ features / pixels / etc. Imagine each input can be classified into $C$ classes Let the ...
Foobar's user avatar
  • 349
1 vote
0 answers
128 views

Explanation of the derivation of the analytical gradient for a SVM?

I'm trying to understand how to derive the analytical gradient for a SVM. I know that in a SVM, the loss function is defined as follows: From this blogpost, I know the full loss for each element in ...
Foobar's user avatar
  • 349
3 votes
1 answer
369 views

How does Generalized Random Forest calculate the gradient of the score function?

The reference is GENERALIZED RANDOM FORESTS by ATHEY, TIBSHIRANI and WAGER (2019). They construct a general algorithm to grow trees and forest for estimation of target parameters that are conditional ...
Dor Leventer's user avatar
5 votes
2 answers
552 views

Why Reparameterization Trick does not work with discrete latent variables?

I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete ...
Curious's user avatar
  • 421
2 votes
0 answers
113 views

Likelihood-ratio gradient estimator in linear dynamical system in python (Jax)

TL;DR I am trying to implement the likelihood-ratio gradient estimator in a linear dynamical system (LDS) with Gaussian transition noise and Gaussian observation noise I am currently using python and ...
Archie42's user avatar
1 vote
0 answers
108 views

Is Gradient Accumulation equivalent to using larger batch sizes?

Gradient accumulation is used to deal with memory limitation by partitioning a large batch size into small chunks. For example, instead of using a batch size of 1024 samples per batch you could use ...
James Kl's user avatar
1 vote
1 answer
2k views

Gradient of a multivariate function numpy

I'm trying to calculate the gradient of multivariate function g using NumPy. g = lambda w: -np.sin(np.pi*np.sum(w**2)) + np.log(np.sum(w**2)) ...
Oguz Aktas's user avatar

1
2 3 4 5