Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

Filter by
Sorted by
Tagged with
1 vote
0 answers
20 views

Why Reparameterization Trick does not work with discrete latent variables?

I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete ...
user avatar
  • 61
1 vote
0 answers
27 views

Likelihood-ratio gradient estimator in linear dynamical system in python (Jax)

TL;DR I am trying to implement the likelihood-ratio gradient estimator in a linear dynamical system (LDS) with Gaussian transition noise and Gaussian observation noise I am currently using python and ...
user avatar
1 vote
0 answers
16 views

Is Gradient Accumulation equivalent to using larger batch sizes?

Gradient accumulation is used to deal with memory limitation by partitioning a large batch size into small chunks. For example, instead of using a batch size of 1024 samples per batch you could use ...
user avatar
0 votes
0 answers
13 views

Deriving Optimizer of Quadratic Loss for Classification

I'm currently considering a binary classification problem where we have data points $X_1,\dots,X_n\in\mathbb{R}^d$ and labels $y_i=\pm1$. I'm using a simple linear model to model $y_i$, and it has the ...
user avatar
  • 157
1 vote
1 answer
89 views

Gradient of a multivariate function numpy

I'm trying to calculate the gradient of multivariate function g using NumPy. g = lambda w: -np.sin(np.pi*np.sum(w**2)) + np.log(np.sum(w**2)) ...
user avatar
0 votes
0 answers
9 views

Which NLP methods use gradient and activation methods?

I am doing a literature review of gradient-based methods for NLP. Yet, apart from linear and logistic regression, I have little knowledge of other methods using the gradient. So I have no knowledge of ...
user avatar
0 votes
0 answers
16 views

Difference between analytic and numeric gradient in Matlab

I am using matlab to solve a optimization problem. When I check the anlaytic and numeric gradient reported by matlab, they are quite different. So I want to ask if there is a mistake in the analytic ...
user avatar
  • 105
0 votes
0 answers
38 views

Difference between forward-mode and reverse-mode automatic differentiation?

I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ...
user avatar
  • 2,031
0 votes
0 answers
32 views

Computing gradients for Gaussian processes using locally periodic kernel

I want to use stochastic gradient descent to find hyperparameters for the locally periodic kernel. The locally periodic kernel is the product of two kernels: the periodic and squared exponential ...
user avatar
1 vote
2 answers
66 views

What is the difference between gradient descent and batch gradient descent? [duplicate]

It seems that batch gradient descent is the traditional gradient descent, except that the objective function is in the form of summation?
user avatar
0 votes
0 answers
73 views

How to Determine Gradient and Hessian for Custom Xgboost Functions

I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python ...
user avatar
  • 51
2 votes
1 answer
50 views

Hard attention derivations

I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ...
user avatar
-1 votes
1 answer
176 views

Do we know the Effects of "RELU Activation Functions" on the Convexity of the Loss Functions in Neural Networks?

Do we know the Effect of "RELU Activation Functions" on the Convexity of the Loss Functions in Neural Networks? I have heard the following argument being made regarding Neural Networks: ...
user avatar
  • 5,718
1 vote
0 answers
328 views

Compute Gradient of Cross Entropy Loss with respect to its logits

I am in the freshman year of my master degree and I have been asked to compute the gradient of Cross Entropy Loss with respect to its logits. I should base the computation on Stanford notes page 4 ...
user avatar
0 votes
1 answer
24 views

Does gradient clipping in a RNN help the network learn the long term dependencies?

So this was asked in one of the exams and I think that gradient clipping does help in learning long term dependencies in RNN but the answer provided to us was "Gradient clipping cannot help with ...
user avatar
2 votes
2 answers
281 views

How GRU solves vanishing gradient

I am learning the GRU model in deep learning and reading this article where details of BPTT are explained. Towards the end the author explained the values of the partial derivative $\frac{\partial h_i}...
user avatar
  • 308
0 votes
0 answers
38 views

Effect of averaging gradients and shuffling (PPO)

In PPO (reinforcement learning algorithm) one often takes large batchsizes like 100000. To do that one averages the gradients 100 minibatches of batchsize 1000. As I understand it is recommended to ...
user avatar
0 votes
0 answers
50 views

Maximum Likelihood estimation of Double Poisson GAS models in R

I am trying to fitting a Bivariate Poisson generalized autoregressive score (GAS) model in R. The model is developed in this paper: https://papers.tinbergen.nl/17062.pdf . I have found a working code ...
user avatar
0 votes
1 answer
54 views

Rank of gradient-of-loss with respect to layer weights in an MLP

The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3: The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch ...
user avatar
  • 3
0 votes
0 answers
20 views

what is the advantage of using Hamilton dynamics in sampling methods? [duplicate]

I am wondering apart form being gradient based sampling methods, what is the advantages of using Hamiltonian MCMC?
user avatar
  • 123
0 votes
1 answer
69 views

How to show that the gradient of the smoothed surrogate loss function leads to perceptron update?

This is about the contents of section 1.2.1 and 1.2.1.1 of the book "Neural Networks and Deep Learning: A Textbook". The link to the sections is here. The question arises from the following ...
user avatar
  • 1
7 votes
2 answers
856 views

In GD-optimisation, if the gradient of the error function is w.r.t to the weights, isn't the target value dropped since it's a lone constant?

Suppose we have the absolute difference as an error function: $\mathit{loss}(w) = |m_x(w) - t|$ where $m_x$ is simply some model with input $x$ and weight setting $w$, and $t$ is the target value. In ...
user avatar
  • 609
0 votes
0 answers
32 views

Why does the Law of the Unconscious Statistician work here for the pathwise estimator

https://arxiv.org/abs/1906.10652 So there are these two parts "Continuous distributions have a simulation property that allows both a direct and an indirect way of drawing samples from them, ...
user avatar
  • 93
1 vote
1 answer
54 views

Derivative of $\nabla_{\theta} f(x, \theta) f(x, \theta)$ (the gradient of the function times the function itself)

I am having troubles computing the derivative of $\nabla_{\theta}f(x, \theta)f(x, \theta) $ (the gradient of the function $f(x, \theta)$ times the function itself) that is \begin{align} D(\nabla_{\...
user avatar
2 votes
0 answers
53 views

Gradient exploding problem in a graph neural network

I have a gradient exploding problem which I couldn't solve after trying for several days. I implemented a custom message passing graph neural network in tensorflow which is used to predict a ...
user avatar
1 vote
0 answers
39 views

Is there a general closed-form formula of the derivates of a feedforward network?

I am looking for a general closed-form formula for the derivatives of a Feed-forward Network with respect to the inputs. Mathematically, we can write: $$ \mathbf{y} = f_{FF}(\mathbf{x}) = \mathbf{W}_{...
user avatar
  • 31
0 votes
0 answers
36 views

Does the existence of gradient in any function necessarily imply the existence of a subgradient at that point?

First , I apologize if the question is not supposed to be here, or if it is off topic for the subjects dealt with in here. I was reading on subgradients, with respect to convex functions in the ...
user avatar
  • 313
0 votes
0 answers
71 views

Mathematical formalism of Gradient Boosting Decision Trees (GBDT) algorithms

I'm trying to better figure out some formalism behind the Gradient Boosting Decision Trees (GBDT) algorithms. Given a dataset $\mathcal{D}$ and a loss function $L : \mathbb{R}^2 \rightarrow \mathbb{R}$...
user avatar
0 votes
0 answers
42 views

Why isn't my gradient descent code converging to solution for GB2 probability distribution?

I'm running gradient descent code in R on an $n$=10,000 test dataset simulating insurance claims records that follow the Generalized Beta of the 2nd Kind ...
user avatar
  • 4,504
2 votes
1 answer
81 views

How to compute the gradient for a GARCH with the package rugarch in R

I am estimating a GARCH(1,1) with external regressors and the package rugarch allows me to do it easily. However, to compute QMLE robust standard errors, I need the ...
user avatar
2 votes
0 answers
48 views

How to Efficiently Finding All Local Maxima in a Large Parameter Space

I am working in 8-D parameter space, where every parameter is on the interval [0, 1]. The number of local maxima in this space and how they are positioned relative to one another is way more ...
user avatar
  • 187
0 votes
0 answers
38 views

Gradient with respect to a probability distribution

I'm trying to understand the worked propositions on this paper (reinforcement learning): https://arxiv.org/abs/2007.02832 The authors formulate this objective: $\hat{g}^*=\arg\max_{\hat{g}\in B}\...
user avatar
0 votes
0 answers
29 views

Derivatives in Vanilla RNNs

I was going through the Vanishing gradients paper by Mikolov ,and just in the beginning of the paper in equations (5) he mentions that Where $x_t$ is the hidden state of an RNN at time $t$, which is ...
user avatar
2 votes
1 answer
101 views

gradient descent in neural network

Given that almost all the activation functions in neural networks are increasing, by the gradient descent rule, all parameters should be updated in the same direction (negative direction). Then how ...
user avatar
  • 85
0 votes
2 answers
250 views

How to do gradient descent when parameter is positive definite matrix

So, suppose I have an objective function $\mathcal{L}(\Sigma)$ where $\Sigma$ is a positive definite matrix. Now, I want to optimize this function using gradient descent. Now, I think if I compute the ...
user avatar
1 vote
2 answers
84 views

Intuition About Gradient Descent Convergence

I know that gradient descent takes steps towards a minimum, but I am having trouble coming up with intuitions about when it will converge. For example, on any given convex function is gradient descent ...
user avatar
  • 101
0 votes
0 answers
32 views

"Double gradient" design for ecotoxicology of particulate substances

We are designing an investigation on effects of microplastic particles (MP) on experimental soil in growth chambers. Question: Is there a significant effect? And if so, what is the form of the ...
user avatar
  • 1
3 votes
1 answer
135 views

Deriving the gradients for Softmax logistic regression classifier

In the softmax logistic regression classifier, we have that $$\textbf{a} = W\textbf{x} + b\\[1ex] \textbf{z} = \text{softmax}(\textbf{a})\\[1ex] L(\textbf{z},\textbf{y}) = -\sum_k \log(z_k)y_k$$ In ...
user avatar
  • 876
0 votes
0 answers
24 views

Computing the Jacobian $J_F$ with $F = h \circ f$

Let $$ f: \mathbb{R}^l \rightarrow{} \mathbb{R}^m\\[.7ex] h: \mathbb{R}^m \rightarrow{} \mathbb{R}^o$$ and let $$F = h \circ f \quad (F : \mathbb{R}^l \rightarrow{} \mathbb{R}^o)$$ I want to compute ...
user avatar
  • 143
0 votes
0 answers
53 views

Is convexity actually necessary for gradient descent?

I understand that the problem that when you are optimizing something using gradient descent, the algorithm might get stuck in a local optimum that isn't global. Otherwise, there are non-convex ...
user avatar
1 vote
0 answers
32 views

A step of gradient descent is upper bounded by a function of gradient norm?

In Appendix A.2 of paper "Maximum a Posteriori Policy Optimisation"(Page 17), the authors say $f$ in Equation $(23)$ is simply a gradient descent update w.r.t. $\theta$ from the original $f$...
user avatar
  • 885
3 votes
1 answer
337 views

What is gradient of the objective function of lasso regression

In LASSO regression we minimise $(Xw-y)^T(Xw-y)+λ \|w \|$. How to find the gradient of its objective function?
user avatar
  • 31
1 vote
1 answer
279 views

Interpretation of Gradient and Hessian for Categorical variables in Gradient Boost

Before searching for split points LightGBM sorts categories withing categorical features by: $\frac{\sum_{i=1}^{n} 1_{x_{i j}=x_{i k}} g_{i}}{\sum_{i=1}^{n} 1_{x_{i j}=x_{i k}} h_{i}}$ Where $g_i$ is ...
user avatar
1 vote
1 answer
34 views

Clarification needed on gradients in backpropagation

I was going through this book "Practical Convolutional Neural Networks" and there under the backpropagation section, it demonstrates calculating the gradient for ...
user avatar
1 vote
0 answers
123 views

How do you find the derivative (gradient) of the general linear regression method?

I am very new to statistical learning (I'm a graduate student in experimental biology with very little exposure to math or statistics) and I'm working my way through Introduction to Statistical ...
user avatar
1 vote
1 answer
49 views

Does the approximation of the gradient of the log-likelihood imply the approximation of the log-likelihood?

Let $ln(p(\mathcal{D};\theta))$ be the log-likelihood function, such that $\mathcal{D}=\{\mathbf{x}_1,\mathbf{x}_2,...,\mathbf{x}_N\}$ is the observed data and $\theta$ is a vector of parameters. Now ...
user avatar
  • 2,956
4 votes
1 answer
55 views

Why do we not use continuously defined losses in NLP?

I understand that various problems in optimization in NLP which do not exist on continuous tasks such as vision, arise in NLP because we do not have continuous data to predict, but one-hot vectors ...
user avatar
  • 43
2 votes
1 answer
466 views

The gradient of neural networks w.r.t one-hot encoded inputs

One-hot encoding as raw inputs for deep learning models can find its applications in many domains, such as bioinformatics, NLP, chemistry and so on. Suppose we trained a neural network $f(x)$ with $x$ ...
user avatar
  • 1,388
4 votes
1 answer
1k views

How Gradient Descent is used for classification with Decision Trees?

I'm not able to see how do we use gradient descent to minimize the loss of binary classification with decision tree. What I understood is that we first have a model (decision tree) that try to predict ...
user avatar
0 votes
0 answers
20 views

Is my understanding and presentation of concept of Gradient Boosting correct?

Initially the model is trained with a training set $\{x_{i}, y_{i}\}_{i=1}^{n}$ by minimizing a differentiable loss function $L(y, F(x))$, and, is initialized with a constant value, \begin{align*} F_{...
user avatar
  • 101