Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

191 questions
Filter by
Sorted by
Tagged with
1 vote
20 views

### Why Reparameterization Trick does not work with discrete latent variables?

I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete ...
1 vote
27 views

### Likelihood-ratio gradient estimator in linear dynamical system in python (Jax)

TL;DR I am trying to implement the likelihood-ratio gradient estimator in a linear dynamical system (LDS) with Gaussian transition noise and Gaussian observation noise I am currently using python and ...
1 vote
16 views

### Is Gradient Accumulation equivalent to using larger batch sizes?

Gradient accumulation is used to deal with memory limitation by partitioning a large batch size into small chunks. For example, instead of using a batch size of 1024 samples per batch you could use ...
13 views

### Deriving Optimizer of Quadratic Loss for Classification

I'm currently considering a binary classification problem where we have data points $X_1,\dots,X_n\in\mathbb{R}^d$ and labels $y_i=\pm1$. I'm using a simple linear model to model $y_i$, and it has the ...
1 vote
89 views

### Gradient of a multivariate function numpy

I'm trying to calculate the gradient of multivariate function g using NumPy. g = lambda w: -np.sin(np.pi*np.sum(w**2)) + np.log(np.sum(w**2)) ...
9 views

### Which NLP methods use gradient and activation methods?

I am doing a literature review of gradient-based methods for NLP. Yet, apart from linear and logistic regression, I have little knowledge of other methods using the gradient. So I have no knowledge of ...
16 views

### Difference between analytic and numeric gradient in Matlab

I am using matlab to solve a optimization problem. When I check the anlaytic and numeric gradient reported by matlab, they are quite different. So I want to ask if there is a mistake in the analytic ...
38 views

### Difference between forward-mode and reverse-mode automatic differentiation?

I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ...
32 views

### Computing gradients for Gaussian processes using locally periodic kernel

I want to use stochastic gradient descent to find hyperparameters for the locally periodic kernel. The locally periodic kernel is the product of two kernels: the periodic and squared exponential ...
1 vote
66 views

### What is the difference between gradient descent and batch gradient descent? [duplicate]

It seems that batch gradient descent is the traditional gradient descent, except that the objective function is in the form of summation?
73 views

### How to Determine Gradient and Hessian for Custom Xgboost Functions

I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python ...
50 views

### Hard attention derivations

I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ...
176 views

### Do we know the Effects of "RELU Activation Functions" on the Convexity of the Loss Functions in Neural Networks?

Do we know the Effect of "RELU Activation Functions" on the Convexity of the Loss Functions in Neural Networks? I have heard the following argument being made regarding Neural Networks: ...
1 vote
328 views

### Compute Gradient of Cross Entropy Loss with respect to its logits

I am in the freshman year of my master degree and I have been asked to compute the gradient of Cross Entropy Loss with respect to its logits. I should base the computation on Stanford notes page 4 ...