Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Proximal gradient descent for “projective” $l1$ term

Proximal gradient descent is employed when you want to find $\min f$ where $f(v)=g(v)+h(v)$, $g$ is smooth and $h$ is not smooth. When $h$ is a lasso term of the form $h(v)=\lambda ||v||_1$, the ...
0
votes
0answers
25 views

Difference between gradient descent and back propagation

So when optimizing the parameters in both logistic regression and Neural networks we use gradient descent. But in logistic regression to show the gradient descent we use the cost function on the y-...
0
votes
0answers
29 views

Initializing network weights to zero

Since my last question on the topic I have tried searching on my own how zero weight initialization impedes learning but I can't quite seem to wrap my head around the concept. The CS231n course notes ...
0
votes
1answer
77 views

Neural ODEs gradient calculation for multiple time steps

I was reading the paper on Neural ODEs (here) and was wondering if anyone could offer some insight on calculation of the gradient of the loss function. If we are only considering 2 time points, $t_0,...
1
vote
0answers
22 views

What neural network loss function to use for multi-class, multi-label classification tasks where labels refer to counts?

I would like to train a neural network to detect which of N classes is present, and in what amounts. In other words each example x has label vector y where y_i >= 0. For example, this could be an ...
5
votes
2answers
97 views
+100

Bayesian interpretation of logistic ridge regression

Most textbooks (also this blog) cover the fact that ridge regression, $$ \hat y = \hat \beta X; \\ \hat \beta = \underset{\beta}{\text{argmin}}\ \ \frac{(y-\beta X)^T(y-\beta X)}{\sigma^2} + \lambda \...
0
votes
0answers
7 views

PPO anti-clip: how to increase update step? [closed]

Batch size 512, discrete action space, tried Adam optimizer with learning rate from 0.0001 to 0.1. Loss doesn't decrease and action probabilities don't change much, e.g. from [1, 0, 0] to [0.9999998, ...
0
votes
0answers
20 views

Does gradient decent happen during the back propagation of a layer or after back propagation is done for all layers?

I'm currently learning the back propagation algorithm for neural network and I need to clear some confusions: From what I understood, during the back propagation algorithm we would calculate the ...
0
votes
0answers
6 views

How do I test my recommender system? [duplicate]

I have created a recommender system based on collaborative filtering with gradient descent. I have completed the training. Now how do I test my recommender system for new users or new items? Here is ...
0
votes
0answers
12 views

What can cause a GNN to diverge?

I'm using A GIN (https://arxiv.org/abs/1810.00826) with a TopK pooling (https://arxiv.org/abs/1905.02850) and an Adam optimizer with some of my own data. Back-propagating works well as the loss ...
1
vote
0answers
17 views

Does the rate of convergence of optimizers matter in deep learning?

In classical optimization, an enormous amount of effort is taken to characterize the rate of convergence of optimization algorithms and designing fast gradient algorithms. You can find tables upon ...
2
votes
1answer
48 views

clarification on back-propagation calculations for a fully connected neural network

I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated. Let's say my fully connected neural network looks like ...
0
votes
0answers
16 views

How to improve Levenberg-Marquardt's method for polynomial curve fitting?

Some weeks ago I started coding the Levenberg-Marquardt algorithm from scratch in Matlab. I'm interested in the polynomial fitting of the data but I haven't been able to achieve the level of accuracy ...
0
votes
1answer
17 views

automatic diffentiation (autograd): when the explicit definition of the gradient function is needed?

In Pytorch and similar machine learning software, the Autograd module computes the gradient of a function without needing to explicit declare the derivative of each single function which composes the ...
0
votes
0answers
5 views

What is the maximum size of weights update in Momentum optimisation?

Given the optimization rules of momemntum gradient descent for weights update of a neural network: $$m\leftarrow \beta m -\eta \nabla_\theta J(\theta)$$ $$ \theta \leftarrow \theta + m$$ where $\...
1
vote
0answers
15 views

Why is breaking symmetry important, when initializating the weights of a Neural Network? [duplicate]

In the beginning of the training process of a Neural Network, it's parameters, for example the weights in a Fully Connected Layer, have to initialized. There is a wide variety of schemes, how you can ...
1
vote
1answer
23 views

Convergence under large set of learning rates

What is the interpretation of a stochastic optimization problem where a gradient descent algorithm is converging under a wide range of learning rate schedules (including ones with quite large initial ...
5
votes
1answer
72 views
+200

Do multiple deep descents exist?

To my knowledge, the phenomenon of double deep descent is still not well understood, but several authors have reported what they call: Model-wise double descent ("double descents" observed ...
3
votes
0answers
21 views

Does gradient descent assume updates of one layer/parameter at a time?

I read the following in "Deep Learning", from Goodfellow et al (Chapter 8, page 313): The gradient tells how to update each parameter, under the assumption that the other layers do not change. In ...
1
vote
1answer
21 views

ANN Cost Function Notation

I have been following This book on the fundamentals of NNs. It is currently outlining the MSE Cost function, and the Notation is tripping me up some. $$ C(w, b) = \dfrac{1}{2n} \sum_x \vert\vert y(x)...
0
votes
0answers
10 views

Different rates learning alpha in gradient descent with multiple variables

I was doing a project, and was trying to get the theta values through the gradient descent method. First i choose a value of 0.01 for alpha and got the values for my theta- 334302, 99411,3267 (all ...
3
votes
0answers
18 views

Benefits of saturation in activation functions

It's known that saturation of activation functions in neural networks leads to vanishing gradients or dead units, so modern practice often avoids them, instead opting for e.g. ReLUs, Leaky ReLUs and ...
0
votes
0answers
12 views

Gradient descent: average gradient vs average forward/backward of mini-batch points

Which is more used in practice? Does it make a difference in computational efficiency and efficacy ? A Compute the gradient on a single point of a mini-batch Average the gradient update the weights ...
0
votes
0answers
12 views

Linear Model with gradient Descent: Adaptive learning rate method

I am analyzing a gradient descent implementation of a linear classifier. Before each gradient update, the learning rate is updated as: ...
1
vote
1answer
14 views

How do you find the minima of a function in python? [closed]

Say we have a quadratic function in x, where the domain of input x is Real Numbers. How can we find the minimum value of the function (output y) in a programming language like python? Immediately ...
0
votes
0answers
15 views

Gradient clipping in high vs low dimensions

Is there a qualitative difference between (l2-norm based) clipping a gradient in a low vs a high dimensional statistical model? In which regime is clipping operation expected to work better?
0
votes
1answer
34 views
2
votes
1answer
29 views

Reinforcement learning using the gradient of expected value doesn't lead to the optimal policy

I'm trying to learn more about reinforcement learning, and I've devised a very simple game as a thought experiment. The game consists of a single turn where the agent plays one of three possible cards....
0
votes
1answer
21 views

Why in general is early stopping a good regularisation technique?

Early stopping means stopping gradient descent when the validation error starts to increase. This is commonly used for neural networks, but can also be used for any model trained by gradient descent, ...
0
votes
1answer
55 views

Lasso Regression

I'm a beginner and hope someone could at least point me a direction on how to write out the gradient for Lasso Regression, thank you so much! $J_{\beta_0,...\beta_m} = \frac{1}{2m}\sum_{i=1}^{m} (y_i ...
1
vote
1answer
19 views

How does having different scales on features make an elliptical contour plot?

I have been taking Andrew Ng's Machine Learning course, and in the lesson on feature scaling's effect on gradient descent, I just can't understand how because of the different scales on the features ...
2
votes
1answer
54 views

Why am I not getting the correct output from my gradient descent algorithm? [closed]

I have started taking online ML classes, and i was introduced to the topic of Gradient Descent, the Prof, himself hadnt shown us himself how to implement it in a programming language, so for fun, i ...
1
vote
0answers
26 views

Normalized steepest descent with nuclear/frobenius norm

In steepest gradient descent, we try to find a local minima to a loss function $f(\cdot)$ by the rule: $x_{t} = x - \alpha \triangledown_{x}f(x)$. I've found in textbooks that often we want to ...
1
vote
0answers
24 views

How to set the tolerance in Gradient descent?

I understand that one solution of setting the number of iterations, is to set it to a large number and then interrupt it when the gradient vector becomes tiny, so tiny that it is smaller than a ...
1
vote
0answers
22 views

BackPropagation and Flatten layer in CNN

everybody. I'm trying to create CNN(Convolutional Neural Network) without frameworks(such as PyTorch,TensorFlow,Keras and so on) on Python. Who don't know or forgot what is exactly CNN is: To ...
2
votes
1answer
36 views

Gradient Bandit Algorithm

I read about the Gradient Bandit Algorithm as a possible solution to the Multi-armed Bandits, and I didn’t understand it. I would be happy if anyone can send me a link to a video, blog post, book, ...
1
vote
0answers
28 views

How to calculate number-of-ops used in the backward pass of the neural net in training phase?

I'm trying to study a basic model like ResNet and how many operations it does and memory usage during backward-pass. For forward pass for layer like 1x1 conv or 3x3 conv, i was able to easily compute ...
1
vote
1answer
33 views

How to decouple weight decay strength and model size?

Consider the neural networks' loss function with the cross entropy term and the $L^2$ weight decay term, which are usually written as: $$E = \frac{1}{N_{samples}} \sum_{i=1}^{N_{samples}} \text{...
0
votes
0answers
19 views

In backprop/gradient descent, why isn't the reciprocal of gradient used when updating weights?

I've seeing almost all tutorials on backprop stating the following for weight update: $$ W_n = W_{n-1} - lr \times L\frac{dL}{dW} $$ Since $\frac{dL}{dW}$ is the influence of $W$ on $L$, and we are ...
0
votes
0answers
30 views

How Hessian matrix is helping in taking big step towards minimization and how is it better than usual Gradient Descent? [duplicate]

I know what Hessian is and $θ:=θ−H^{-1}f′(θ)$ this relation too from Newton Raphson but what i dont understand is how Hessian is really helping with big step and also how is this efficient in ...
0
votes
0answers
9 views

Reversing a regression with polynomial features

For an assignment, I have fit a model with polynomial (quadratic) generated features: $$Y = \Theta X'$$ There are only 5 original features $x$, but after transforming to quadratic features $x'$ ...
0
votes
0answers
13 views

Using gradient descent to find the ground truth pdf

I have a function $I_d(x)$ which defined over a plane. I could simulate the values of this function at different points. I have a ground truth probability density vector $p({\bf x})=(p_1(x),...,p_d(x))...
1
vote
1answer
24 views

How to adapt the equations for stochastic gradient descent for batch gradient descent for neural networks?

I’m following along this lecture on neural networks. The professor derives equations for the gradient of $e(w)$: $\frac{\partial e(w)}{w_{ij}^l}$ for every $w_{ij}^l$ where $e(w)=e(h(x_n),y_n)$ is the ...
0
votes
2answers
48 views

Why do earlier hidden layers learn slower?

I'm reading chapter 5 of Nielsen's textbook about vanishing gradients. He states: In at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden ...
0
votes
0answers
7 views

gradient computation in Nesterov momentum

I am reading deeplearningbook by Ian Goodfellow et al. And I have a question about gradient computation in Nesterov momentum. There are two pages from this book, which describe Nesterov momentum: ...
0
votes
0answers
16 views

Investigator efficiency computation

I need a little help with inferring ground truth and investigation capabilities from data. I have an incomplete matrix consisting of binary decisions taken by investigators on documents i.e. for every ...
0
votes
0answers
60 views

Valid use of differentiable almost everywhere functions, like hinge loss, in gradient optimization/learners, like SciKit-Learn's SGDClassifier?

So, my abstract question is: is it valid (in the sense that stable convergence is roughly expected) to use functions that are differentiable almost everywhere in the practical application of ...
0
votes
0answers
14 views

Reference request for normalized gradient descent

Can someone introduce a good article/textbook explaining variants of the gradient descent method? In particular, I am interested in the normalized gradient descent where one works with $\frac{\nabla f}...
1
vote
0answers
36 views

Can gradient descent work iteratively instead of simultaneously?

When multivariate gradient descent is updating weight of single feature, why doesn't it project this new weight when updating weights of next features? Let's say we have this example of gradient ...
0
votes
1answer
30 views

How to derive the gradient of the reparameterized score function estimator?

In the paper Evolution Strategies as a Scalable Alternative to Reinforcement Learning, the authors derive the following gradient of the score function estimator $$ \begin{align} \nabla_\psi\mathbb E_{...

1
2 3 4 5
15