Questions tagged [gradient]
Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.
219
questions
3
votes
1
answer
57
views
Can XGBoost learn more complicated interactions/features?
For a set of features {a, b, c, d . . . n}, XGBoost can easily learn, say, a*d. In practice can it also effectively learn a/c? Or (a + b + c + 2)/d? Or (c^(2d))/(b^a)? I'd imagine some of this depends ...
- 151
1
vote
0
answers
32
views
Gradient of Gaussian Process Regressor
I have a data ((x,y),f) that I am fitting using Gaussian Process Regression in Python's sklearn package. The posterior mean of the GP is essentially my output with an associated error. Based on either ...
- 11
4
votes
2
answers
391
views
What is the meaning of "SGD scales the gradient uniformly in all directions"?
I'm really newbie about neural network and optimization. When I read the references, I found this journal Wang et al 2018.
The journal stated:
One disadvantage of SGD
is that it scales the gradient ...
- 117
2
votes
1
answer
49
views
Exploding/Vanishing gradients deeper understanding
I'm trying to gain deper understanding of the logic behind vanishing and exploding gradients. Most sources I've come across explain the problem by saying that when the weights become too small, the ...
- 23
1
vote
0
answers
33
views
Closed form expression for the gradient of a fully connected neural network with respect to its parameters
A real-valued feedforward/fully connected neural network with activation function $\sigma : x\in \mathbb R \mapsto \max \{0,x\}\equiv \text{ReLU}(x)$ can formally be seen as a function $f_\theta :\...
0
votes
0
answers
20
views
How do you find the gradients of weights and biases in neural network during back propagation?
I have been trying to create a neural network from scratch.
I have been trying to calculate the gradients of the weights and biases of the neural network by watching videos and reading papers, but ...
0
votes
0
answers
35
views
How to measure the noise in gradients of deep neural networks
I'm dealing with a problem related to explainable AI. The short version of the issue I have is that I've observed some interesting differences between large and small BERT models and I suspect that ...
- 11
1
vote
0
answers
50
views
Explainable AI - Noise in gradients and embeddings of large language models
I am doing experiments related to explainable AI. I have two BERT models - the standard bert-base-cased and a distilled ...
- 11
1
vote
0
answers
50
views
Neural network parameters dependency vs gradient descent [duplicate]
Very deep models involve the composition of several functions or
layers. The gradient tells how to update each parameter, under the
assumption that the other layers do not change. In practice, we ...
- 433
1
vote
0
answers
34
views
Which training instances are being subsampled in each iteration of XGBoost?
The subsample option in XGBoost is described here as follows:
Subsample ratio of the training instances. Setting it to 0.5 ...
- 11
0
votes
0
answers
26
views
Vanishing Gradient Problem: What is the cause from a Data perspective? [duplicate]
My question:
I know there exists a lot of information about what causes the vanishing gradient from a computational standpoint. Ie due to way the RNN is trained by backpropagation [...]. Why do RNNs ...
- 101
2
votes
0
answers
60
views
Accelerate the fitting of an ECM-GARCH model by computing MLE gradient numerically?
I'm trying to fit an ECM model with variance following a GARCH-DCC model (GARCH with dynamic cross correlation). It has 16 parameters for 2 assets (ECM : 4 gammas, 2 lambda, GARCH: 2 alphas, 2 beta, 2 ...
- 240
1
vote
0
answers
45
views
Doubts about Gradient and Matrices derivatives
Studying NN and I wanted to grasp from scratch the theory of gradient descent and matrices derivatives, so I took a simple scenario and tried to apply gradient ascent and see if everything made sense.
...
- 317
0
votes
0
answers
24
views
Finding the gradient for logistic regression using sign outputs
In logistic regression, if we use the sign outputs, such that $y \in \{-1,1\}$, we have that the loss function is given by (from [here])
$$L(y,\beta^Tx)=\log(1+\exp{(-y\cdot \beta^Tx}))$$
In this case,...
- 105
0
votes
0
answers
50
views
Why are ranking loss functions hard to optimize?
Currently I am learning about ranking loss functions, in particular the normalized cumulative gain, which can be written as:
$NDCG = \frac{DCG}{IDCG}$
where DCG is defined as
$DCG = \sum_{i=1}^N \frac{...
- 505
2
votes
0
answers
68
views
Why doesn't Logloss match Similarity score in this xgboost example?
I am trying to get direct connection between Gain and Logloss for XGBoost. It looks to me that in Xgboost paper formula 6:
for a model with depth 1 and number of trees=1 this formula contains ...
- 33
0
votes
0
answers
23
views
XGBoost logloss calculation doesnt match documentation
XGBoost Logloss formula from doc https://xgboost.readthedocs.io/en/stable/tutorials/model.html
is $\sum_i^n l(y_i, \hat{y}_i) + \sum_{k=1}^K \omega(f_k)$.
However when I calculate it with hands in ...
- 33
0
votes
0
answers
15
views
Neural Network training as non stationary stateless continuous reinforcement learning problem
Say I have a neural network denoted as f(\theta), and we want to optimize $\theta$.
What I thought is that $\theta$ can be seen as an action sampled from a ...
1
vote
1
answer
108
views
Gain vs Loss in terms of selecting best leaf split value
In the XGBoost Documentation they specify the Gain term as:
\begin{equation}
Gain=\frac{1}{2} \left[ \frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda}- \frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\...
- 33
3
votes
1
answer
54
views
Inference of Gradient Boosting on test istance
According to my professor, Gradient Boosting can be done using the following algorithm:
Now, I do not really understand the inference part of that algorithm.
Why cannot we not simply return $F^{(K)}(...
- 505
6
votes
0
answers
728
views
Why do we minimise a cost function instead of maximising an equivalent? [duplicate]
I don't really understand why we minimise a cost function for gradient descent. Why don't we try to have something like a gradient 'climb', where we maximise some function?
Is it due to convention, or ...
- 61
0
votes
0
answers
250
views
Training a physics informed neural network (PINN) in Julia using numerical gradient approximations
i am currently working on a small project which involves solving a pendulum differential equation
$$a \ddot{x} + b \dot{x} + x = 0$$
The idea is to use a physics informed neural network which has two ...
- 321
0
votes
0
answers
24
views
Fast renormalization of distributions
Say I have a matrix probs $\in N\times4$ where each row is a categorical distribution.
However I have also a mask $\in N\times4$ that is meant to remove actions that are not available.
What I used to ...
4
votes
2
answers
245
views
Why there is high variance of gradients estimated in the short directions in regression?
I was trying to understand Ridge Regression and came across the following excerpt from Hastie et al. in The Elements of Statistical Learning (section 3.4.1, Page 67):
If we consider fitting a linear ...
- 43
3
votes
1
answer
319
views
Gradient and Hessian of loss function
I'm trying to clear up the calculation of the gradient and Hessian of a loss function in an article that I am currently reading. The loss function is given by
$$\ell(\beta)=\sum_{i=1}^{N} e^{-y_{i}{{x}...
- 129
1
vote
0
answers
43
views
Is there anyway to calculate the integral of a trace? [closed]
I would like to calculate the integral of a scalar function as follows:
$$f(x)=\mathrm{tr}((\mathbf{A}x+\mathbf{B})^{-1}\mathbf{B}),$$
where $\mathbf{A}$ and $\mathbf{B}$ are two $n\times n$ positive-...
- 21
1
vote
0
answers
34
views
How to merge 2 losses in a reasonable ratio
my question is pretty basic, but I can't find many resources online about this.
Say we have two losses for a model, for example a pix2pix GAN, which for those who are not familiar with it, for the ...
1
vote
1
answer
66
views
Making sense of the Gain term in Gradient tree boosting
In the XGBoost Documentation they specify the Gain term as
\begin{equation}
Gain=\frac{1}{2} \left[ \frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda}- \frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\...
- 125
2
votes
1
answer
191
views
Why do we regularize large gradients corresponding to large errors?
While reviewing some scientific blogs, I found them recommending using gradient clipping for large error gradients. However, intuitively one would think that when model predictions are completely off, ...
- 500
1
vote
1
answer
67
views
Lowering the weight of particular features in a neural network?
Given sample data $x$, we hypothesize that some features (i.e. dimensions) of $x$ will generalize well, while others will generalize poorly. For example, when predicting medical diagnosis, age and ...
- 121
0
votes
1
answer
204
views
Need for reparameterization trick in RL (and others)?
This is a multi-fold question that has a number of closely related questions; that is why I will pose them all here, instead of separate questions.
In RL you have a parameterized policy that dictates ...
- 135
1
vote
0
answers
7
views
Is backprop computed for every element in the minibatch and then averaged for every weight?
I'm trying to fully understand backpropagation by computing it by hand.
Often is cited that is just an alternation of the derivative of the preactivation and the derivative of the activation, however, ...
4
votes
1
answer
723
views
Can Poisson deviance be used to evaluate models that use loss functions other than Poisson? (Such as MSE)
I am currently doing a a study on emergency department utilization rates at various geography levels. Especially of interest, are tree-based approaches to this analysis - namely random forest and GBMs....
- 143
1
vote
0
answers
210
views
How does the full derivative of softmax + cross entropy have the correct dimensions?
The blog post the softmax function and its derivative explains the following:
Imagine that each input has $N$ features / pixels / etc.
Imagine each input can be classified into $C$ classes
Let the ...
- 247
1
vote
0
answers
70
views
Explanation of the derivation of the analytical gradient for a SVM?
I'm trying to understand how to derive the analytical gradient for a SVM.
I know that in a SVM, the loss function is defined as follows:
From this blogpost, I know the full loss for each element in ...
- 247
3
votes
1
answer
261
views
How does Generalized Random Forest calculate the gradient of the score function?
The reference is GENERALIZED RANDOM FORESTS by ATHEY, TIBSHIRANI and WAGER (2019). They construct a general algorithm to grow trees and forest for estimation of target parameters that are conditional ...
- 31
5
votes
2
answers
388
views
Why Reparameterization Trick does not work with discrete latent variables?
I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete ...
- 411
2
votes
0
answers
85
views
Likelihood-ratio gradient estimator in linear dynamical system in python (Jax)
TL;DR
I am trying to implement the likelihood-ratio gradient estimator in a linear dynamical system (LDS) with Gaussian transition noise and Gaussian observation noise
I am currently using python and ...
- 21
1
vote
0
answers
77
views
Is Gradient Accumulation equivalent to using larger batch sizes?
Gradient accumulation is used to deal with memory limitation by partitioning a large batch size into small chunks. For example, instead of using a batch size of 1024 samples per batch you could use ...
- 11
1
vote
1
answer
2k
views
Gradient of a multivariate function numpy
I'm trying to calculate the gradient of multivariate function g using NumPy.
g = lambda w: -np.sin(np.pi*np.sum(w**2)) + np.log(np.sum(w**2))
...
- 13
0
votes
0
answers
13
views
Which NLP methods use gradient and activation methods?
I am doing a literature review of gradient-based methods for NLP. Yet, apart from linear and logistic regression, I have little knowledge of other methods using the gradient. So I have no knowledge of ...
1
vote
0
answers
259
views
Difference between forward-mode and reverse-mode automatic differentiation?
I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ...
- 2,211
1
vote
2
answers
1k
views
What is the difference between gradient descent and batch gradient descent? [duplicate]
It seems that batch gradient descent is the traditional gradient descent, except that the objective function is in the form of summation?
- 119
1
vote
1
answer
1k
views
How to Determine Gradient and Hessian for Custom Xgboost Functions?
I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python ...
- 61
2
votes
1
answer
81
views
Hard attention derivations
I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ...
- 47
-1
votes
1
answer
788
views
Do we know the Effects of "RELU Activation Functions" on the Convexity of the Loss Functions in Neural Networks?
Do we know the Effect of "RELU Activation Functions" on the Convexity of the Loss Functions in Neural Networks?
I have heard the following argument being made regarding Neural Networks:
...
1
vote
0
answers
1k
views
Compute Gradient of Cross Entropy Loss with respect to its logits
I am in the freshman year of my master degree and I have been asked to compute the gradient of Cross Entropy Loss with respect to its logits. I should base the computation on Stanford notes page 4 ...
- 123
0
votes
1
answer
383
views
Does gradient clipping in a RNN help the network learn the long term dependencies?
So this was asked in one of the exams and I think that gradient clipping does help in learning long term dependencies in RNN but the answer provided to us was "Gradient clipping cannot help with ...
- 195
4
votes
2
answers
2k
views
How GRU solves vanishing gradient
I am learning the GRU model in deep learning and reading this article where details of BPTT are explained. Towards the end the author explained the values of the partial derivative $\frac{\partial h_i}...
- 330
1
vote
1
answer
347
views
Rank of gradient-of-loss with respect to layer weights in an MLP
The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3:
The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch ...
- 13