Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

219 questions
Filter by
Sorted by
Tagged with
57 views

For a set of features {a, b, c, d . . . n}, XGBoost can easily learn, say, a*d. In practice can it also effectively learn a/c? Or (a + b + c + 2)/d? Or (c^(2d))/(b^a)? I'd imagine some of this depends ...
1 vote
32 views

### Gradient of Gaussian Process Regressor

I have a data ((x,y),f) that I am fitting using Gaussian Process Regression in Python's sklearn package. The posterior mean of the GP is essentially my output with an associated error. Based on either ...
391 views

### What is the meaning of "SGD scales the gradient uniformly in all directions"?

I'm really newbie about neural network and optimization. When I read the references, I found this journal Wang et al 2018. The journal stated: One disadvantage of SGD is that it scales the gradient ...
49 views

I'm trying to gain deper understanding of the logic behind vanishing and exploding gradients. Most sources I've come across explain the problem by saying that when the weights become too small, the ...
1 vote
33 views

68 views

### Why doesn't Logloss match Similarity score in this xgboost example?

I am trying to get direct connection between Gain and Logloss for XGBoost. It looks to me that in Xgboost paper formula 6: for a model with depth 1 and number of trees=1 this formula contains ...
23 views

### XGBoost logloss calculation doesnt match documentation

XGBoost Logloss formula from doc https://xgboost.readthedocs.io/en/stable/tutorials/model.html is $\sum_i^n l(y_i, \hat{y}_i) + \sum_{k=1}^K \omega(f_k)$. However when I calculate it with hands in ...
15 views

### Neural Network training as non stationary stateless continuous reinforcement learning problem

Say I have a neural network denoted as f(\theta), and we want to optimize $\theta$. What I thought is that $\theta$ can be seen as an action sampled from a ...
1 vote
108 views

### Gain vs Loss in terms of selecting best leaf split value

In the XGBoost Documentation they specify the Gain term as: \begin{equation} Gain=\frac{1}{2} \left[ \frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda}- \frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\...
According to my professor, Gradient Boosting can be done using the following algorithm: Now, I do not really understand the inference part of that algorithm. Why cannot we not simply return $F^{(K)}(... 6 votes 0 answers 728 views ### Why do we minimise a cost function instead of maximising an equivalent? [duplicate] I don't really understand why we minimise a cost function for gradient descent. Why don't we try to have something like a gradient 'climb', where we maximise some function? Is it due to convention, or ... 0 votes 0 answers 250 views ### Training a physics informed neural network (PINN) in Julia using numerical gradient approximations i am currently working on a small project which involves solving a pendulum differential equation $$a \ddot{x} + b \dot{x} + x = 0$$ The idea is to use a physics informed neural network which has two ... 0 votes 0 answers 24 views ### Fast renormalization of distributions Say I have a matrix probs$\in N\times4$where each row is a categorical distribution. However I have also a mask$\in N\times4$that is meant to remove actions that are not available. What I used to ... 4 votes 2 answers 245 views ### Why there is high variance of gradients estimated in the short directions in regression? I was trying to understand Ridge Regression and came across the following excerpt from Hastie et al. in The Elements of Statistical Learning (section 3.4.1, Page 67): If we consider fitting a linear ... 3 votes 1 answer 319 views ### Gradient and Hessian of loss function I'm trying to clear up the calculation of the gradient and Hessian of a loss function in an article that I am currently reading. The loss function is given by $$\ell(\beta)=\sum_{i=1}^{N} e^{-y_{i}{{x}... 1 vote 0 answers 43 views ### Is there anyway to calculate the integral of a trace? [closed] I would like to calculate the integral of a scalar function as follows:$$f(x)=\mathrm{tr}((\mathbf{A}x+\mathbf{B})^{-1}\mathbf{B}),$$where$\mathbf{A}$and$\mathbf{B}$are two$n\times n$positive-... 1 vote 0 answers 34 views ### How to merge 2 losses in a reasonable ratio my question is pretty basic, but I can't find many resources online about this. Say we have two losses for a model, for example a pix2pix GAN, which for those who are not familiar with it, for the ... 1 vote 1 answer 66 views ### Making sense of the Gain term in Gradient tree boosting In the XGBoost Documentation they specify the Gain term as \begin{equation} Gain=\frac{1}{2} \left[ \frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda}- \frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\... 2 votes 1 answer 191 views ### Why do we regularize large gradients corresponding to large errors? While reviewing some scientific blogs, I found them recommending using gradient clipping for large error gradients. However, intuitively one would think that when model predictions are completely off, ... 1 vote 1 answer 67 views ### Lowering the weight of particular features in a neural network? Given sample data$x$, we hypothesize that some features (i.e. dimensions) of$x$will generalize well, while others will generalize poorly. For example, when predicting medical diagnosis, age and ... 0 votes 1 answer 204 views ### Need for reparameterization trick in RL (and others)? This is a multi-fold question that has a number of closely related questions; that is why I will pose them all here, instead of separate questions. In RL you have a parameterized policy that dictates ... 1 vote 0 answers 7 views ### Is backprop computed for every element in the minibatch and then averaged for every weight? I'm trying to fully understand backpropagation by computing it by hand. Often is cited that is just an alternation of the derivative of the preactivation and the derivative of the activation, however, ... 4 votes 1 answer 723 views ### Can Poisson deviance be used to evaluate models that use loss functions other than Poisson? (Such as MSE) I am currently doing a a study on emergency department utilization rates at various geography levels. Especially of interest, are tree-based approaches to this analysis - namely random forest and GBMs.... 1 vote 0 answers 210 views ### How does the full derivative of softmax + cross entropy have the correct dimensions? The blog post the softmax function and its derivative explains the following: Imagine that each input has$N$features / pixels / etc. Imagine each input can be classified into$C$classes Let the ... 1 vote 0 answers 70 views ### Explanation of the derivation of the analytical gradient for a SVM? I'm trying to understand how to derive the analytical gradient for a SVM. I know that in a SVM, the loss function is defined as follows: From this blogpost, I know the full loss for each element in ... 3 votes 1 answer 261 views ### How does Generalized Random Forest calculate the gradient of the score function? The reference is GENERALIZED RANDOM FORESTS by ATHEY, TIBSHIRANI and WAGER (2019). They construct a general algorithm to grow trees and forest for estimation of target parameters that are conditional ... 5 votes 2 answers 388 views ### Why Reparameterization Trick does not work with discrete latent variables? I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete ... 2 votes 0 answers 85 views ### Likelihood-ratio gradient estimator in linear dynamical system in python (Jax) TL;DR I am trying to implement the likelihood-ratio gradient estimator in a linear dynamical system (LDS) with Gaussian transition noise and Gaussian observation noise I am currently using python and ... 1 vote 0 answers 77 views ### Is Gradient Accumulation equivalent to using larger batch sizes? Gradient accumulation is used to deal with memory limitation by partitioning a large batch size into small chunks. For example, instead of using a batch size of 1024 samples per batch you could use ... 1 vote 1 answer 2k views ### Gradient of a multivariate function numpy I'm trying to calculate the gradient of multivariate function g using NumPy. g = lambda w: -np.sin(np.pi*np.sum(w**2)) + np.log(np.sum(w**2)) ... 0 votes 0 answers 13 views ### Which NLP methods use gradient and activation methods? I am doing a literature review of gradient-based methods for NLP. Yet, apart from linear and logistic regression, I have little knowledge of other methods using the gradient. So I have no knowledge of ... 1 vote 0 answers 259 views ### Difference between forward-mode and reverse-mode automatic differentiation? I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ... 1 vote 2 answers 1k views ### What is the difference between gradient descent and batch gradient descent? [duplicate] It seems that batch gradient descent is the traditional gradient descent, except that the objective function is in the form of summation? 1 vote 1 answer 1k views ### How to Determine Gradient and Hessian for Custom Xgboost Functions? I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python ... 2 votes 1 answer 81 views ### Hard attention derivations I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ... -1 votes 1 answer 788 views ### Do we know the Effects of "RELU Activation Functions" on the Convexity of the Loss Functions in Neural Networks? Do we know the Effect of "RELU Activation Functions" on the Convexity of the Loss Functions in Neural Networks? I have heard the following argument being made regarding Neural Networks: ... 1 vote 0 answers 1k views ### Compute Gradient of Cross Entropy Loss with respect to its logits I am in the freshman year of my master degree and I have been asked to compute the gradient of Cross Entropy Loss with respect to its logits. I should base the computation on Stanford notes page 4 ... 0 votes 1 answer 383 views ### Does gradient clipping in a RNN help the network learn the long term dependencies? So this was asked in one of the exams and I think that gradient clipping does help in learning long term dependencies in RNN but the answer provided to us was "Gradient clipping cannot help with ... 4 votes 2 answers 2k views ### How GRU solves vanishing gradient I am learning the GRU model in deep learning and reading this article where details of BPTT are explained. Towards the end the author explained the values of the partial derivative$\frac{\partial h_i}...
The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3: The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch ...