Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
1
vote
1answer
21 views

Why there is theta in index of gradient symbol in gradient descent update formula for MAML?

In this MAML paper, they use following formula of gradient descent update (see page 3, algorithm 1): $$ \varTheta '\ =\varTheta \ −\ \alpha \nabla _{\varTheta }\mathcal{L}_{\mathcal{T}_{i}}( f_{\...
1
vote
0answers
20 views

Relation between test and train error with gradient descent iterates

My question is about establishing an inequality between population error and expected training error (i.e, expected training error < population error) for a model trained with gradient descent on a ...
1
vote
0answers
27 views

Stopped by zero step from line search - R stops optimization early

I am trying to minimize an objective function, $J(\theta)$, with respect to $\theta$, a 19-dimensional parameter vector. $J(\theta)$ is a smooth nonlinear function so I have tried various gradient-...
0
votes
0answers
21 views

Gradient Descent for changing regression coefficients to minimize error

I recently saw someone input their existing linear regression model into a script that used Adagrad to "tweak" the model's coefficients in order to slightly minimize the model's error. I've ...
3
votes
0answers
50 views

How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
1
vote
0answers
36 views

Why are we interested in gradient with respect to input?

I am learning about sampling methods for Deep Embedding Learning. I was reading an article named: "Sampling Matters in Deep Embedding Learning" (https://arxiv.org/abs/1706.07567). In the ...
3
votes
0answers
36 views

Are there algorithm for finding a minimum of a separately convex function (i.e., $f(x,y)$ convex in $x$ and $y$ but not in $(x,y)$)

Suppose that $f: \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$ is separatly convex function. That is, for a fixed $y$ the mapping $x \to f(x,y)$ is convex and for a fixed $x$ the mapping $y \to f(...
3
votes
0answers
70 views

What are the state of the art optimization methods for neural networks?

Neural networks are usually trained with first order gradient methods and it's variations such as: batch gradient descent, stochastic gradient descent, momentum based gradient and so on.. However ...
0
votes
0answers
67 views

Why not use line search in conjunction with stochastic gradient descent?

I'm familiar with numerical optimization in Engineering context. I have taken several graduate level engineering optimization and operations research courses. I'm beginning to learn machine learning. ...
1
vote
0answers
35 views

Various Methods to Calculate Linear Regression [duplicate]

I have just started learning Machine Learning and one of the very first topics that I have encountered in this venture is Simple Linear Regression. From Andrew Ng's course, I have learned to perform ...
2
votes
1answer
188 views

Mean Square Error and Gradient Descent

I am trying to learn gradient descent and in the course of so I am trying to find the optimal m and c value for my model, for $y=mx+c$ For that, I have plotted the MSE using the below code in python <...
1
vote
1answer
21 views

What's the meaning of these notations in the cost fuction?

Except for the summation, I'm having a hard time figuring out the meaning of these notations. As I assume this is generic and the context is not that necessary, I'm here asking for help. (also, is ...
2
votes
2answers
173 views

Understanding cross entropy loss [closed]

The formula for cross entropy loss is this: $$-\sum_iy_i \ln\left(\hat{y}_i\right).$$ My question is, what is the minimum and maximum value for cross entropy loss, given that there is a negative sign ...
1
vote
1answer
219 views

Why divide the sample size in minibatch gradient descent

Take linear regression for instance the loss is usually define as $L = \frac{1}{2N}\sum_{i = 1}^{N}(Y_i - WX_i)^2$. Two here, as I understand, is to make the derivative look nice. But I am not sure ...
1
vote
1answer
30 views

Does gradient descent work for tabular Q learning?

Suppose I have a tabular Q learning problem such as grid-world. Let our loss be defined as, $$\hat{L}(Q)=0.5(Q(s,a)-(r+\gamma\max_{a'}{Q(s',a')}))^2$$ Then $Q_{k+1}(s,a) = Q_k(s,a) - \eta \nabla \hat {...
2
votes
0answers
94 views

Mini-Batch Gradient Descent - Why does sampling with replacement work?

When sampling the data, either one at a time (as in online learning), or in mini-batches, there exist gradient descent methods which sample with replacement and without replacement. For Mini-Batch ...
1
vote
0answers
75 views

What is wrong with my approach on a custom way of creating Gabor-filter convolution kernels?

Disclosure: I am not a prominent mathematician (current bachelor student) like others on this website and my approach has been mostly pragmatic. Please do tell me if I can improve the formulation of ...
1
vote
0answers
24 views

Momentum vs adaptive step methods

My understanding is that: With momentum, one can avoid e.g. "zig-zags" during gradient decent by averaging gradients to determine a better direction of descent. With adaptive step size ...
0
votes
0answers
18 views

Proximal gradient descent for “projective” $l1$ term

Proximal gradient descent is employed when you want to find $\min f$ where $f(v)=g(v)+h(v)$, $g$ is smooth and $h$ is not smooth. When $h$ is a lasso term of the form $h(v)=\lambda ||v||_1$, the ...
0
votes
0answers
58 views

Initializing network weights to zero

Since my last question on the topic I have tried searching on my own how zero weight initialization impedes learning but I can't quite seem to wrap my head around the concept. The CS231n course notes ...
0
votes
1answer
134 views

Neural ODEs gradient calculation for multiple time steps

I was reading the paper on Neural ODEs (here) and was wondering if anyone could offer some insight on calculation of the gradient of the loss function. If we are only considering 2 time points, $t_0,...
3
votes
0answers
373 views

What neural network loss function to use for multi-class, multi-label classification tasks where labels refer to counts?

I would like to train a neural network to detect which of N classes is present, and in what amounts. In other words each example x has label vector y where y_i >= 0. For example, this could be an ...
6
votes
2answers
332 views

Bayesian interpretation of logistic ridge regression

Most textbooks (also this blog) cover the fact that ridge regression, $$ \hat y = \hat \beta X; \\ \hat \beta = \underset{\beta}{\text{argmin}}\ \ \frac{(y-\beta X)^T(y-\beta X)}{\sigma^2} + \lambda \...
0
votes
0answers
22 views

Does gradient decent happen during the back propagation of a layer or after back propagation is done for all layers?

I'm currently learning the back propagation algorithm for neural network and I need to clear some confusions: From what I understood, during the back propagation algorithm we would calculate the ...
0
votes
0answers
7 views

How do I test my recommender system? [duplicate]

I have created a recommender system based on collaborative filtering with gradient descent. I have completed the training. Now how do I test my recommender system for new users or new items? Here is ...
0
votes
0answers
22 views

What can cause a GNN to diverge?

I'm using A GIN (https://arxiv.org/abs/1810.00826) with a TopK pooling (https://arxiv.org/abs/1905.02850) and an Adam optimizer with some of my own data. Back-propagating works well as the loss ...
1
vote
0answers
73 views

Does the rate of convergence of optimizers matter in deep learning?

In classical optimization, an enormous amount of effort is taken to characterize the rate of convergence of optimization algorithms and designing fast gradient algorithms. You can find tables upon ...
3
votes
1answer
93 views

clarification on back-propagation calculations for a fully connected neural network

I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated. Let's say my fully connected neural network looks like ...
0
votes
0answers
69 views

How to improve Levenberg-Marquardt's method for polynomial curve fitting?

Some weeks ago I started coding the Levenberg-Marquardt algorithm from scratch in Matlab. I'm interested in the polynomial fitting of the data but I haven't been able to achieve the level of accuracy ...
0
votes
1answer
28 views

automatic diffentiation (autograd): when the explicit definition of the gradient function is needed?

In Pytorch and similar machine learning software, the Autograd module computes the gradient of a function without needing to explicit declare the derivative of each single function which composes the ...
0
votes
1answer
36 views

What is the maximum size of weights update in Momentum optimisation?

Given the optimization rules of momemntum gradient descent for weights update of a neural network: $$m\leftarrow \beta m -\eta \nabla_\theta J(\theta)$$ $$ \theta \leftarrow \theta + m$$ where $\...
1
vote
0answers
28 views

Why is breaking symmetry important, when initializating the weights of a Neural Network? [duplicate]

In the beginning of the training process of a Neural Network, it's parameters, for example the weights in a Fully Connected Layer, have to initialized. There is a wide variety of schemes, how you can ...
1
vote
1answer
30 views

Convergence under large set of learning rates

What is the interpretation of a stochastic optimization problem where a gradient descent algorithm is converging under a wide range of learning rate schedules (including ones with quite large initial ...
10
votes
1answer
148 views

Do multiple deep descents exist?

To my knowledge, the phenomenon of double deep descent is still not well understood, but several authors have reported what they call: Model-wise double descent ("double descents" observed ...
4
votes
1answer
75 views

Does gradient descent assume updates of one layer/parameter at a time?

I read the following in "Deep Learning", from Goodfellow et al (Chapter 8, page 313): The gradient tells how to update each parameter, under the assumption that the other layers do not change. In ...
1
vote
1answer
59 views

ANN Cost Function Notation

I have been following This book on the fundamentals of NNs. It is currently outlining the MSE Cost function, and the Notation is tripping me up some. $$ C(w, b) = \dfrac{1}{2n} \sum_x \vert\vert y(x)...
0
votes
0answers
16 views

Different rates learning alpha in gradient descent with multiple variables

I was doing a project, and was trying to get the theta values through the gradient descent method. First i choose a value of 0.01 for alpha and got the values for my theta- 334302, 99411,3267 (all ...
3
votes
0answers
69 views

Benefits of saturation in activation functions

It's known that saturation of activation functions in neural networks leads to vanishing gradients or dead units, so modern practice often avoids them, instead opting for e.g. ReLUs, Leaky ReLUs and ...
0
votes
0answers
18 views

Gradient descent: average gradient vs average forward/backward of mini-batch points

Which is more used in practice? Does it make a difference in computational efficiency and efficacy ? A Compute the gradient on a single point of a mini-batch Average the gradient update the weights ...
0
votes
0answers
14 views

Linear Model with gradient Descent: Adaptive learning rate method

I am analyzing a gradient descent implementation of a linear classifier. Before each gradient update, the learning rate is updated as: ...
1
vote
1answer
105 views

How do you find the minima of a function in python? [closed]

Say we have a quadratic function in x, where the domain of input x is Real Numbers. How can we find the minimum value of the function (output y) in a programming language like python? Immediately ...
0
votes
0answers
16 views

Gradient clipping in high vs low dimensions

Is there a qualitative difference between (l2-norm based) clipping a gradient in a low vs a high dimensional statistical model? In which regime is clipping operation expected to work better?
1
vote
1answer
56 views
2
votes
1answer
41 views

Reinforcement learning using the gradient of expected value doesn't lead to the optimal policy

I'm trying to learn more about reinforcement learning, and I've devised a very simple game as a thought experiment. The game consists of a single turn where the agent plays one of three possible cards....
1
vote
1answer
50 views

Why in general is early stopping a good regularisation technique?

Early stopping means stopping gradient descent when the validation error starts to increase. This is commonly used for neural networks, but can also be used for any model trained by gradient descent, ...
0
votes
1answer
474 views

Lasso Regression

I'm a beginner and hope someone could at least point me a direction on how to write out the gradient for Lasso Regression, thank you so much! $J_{\beta_0,...\beta_m} = \frac{1}{2m}\sum_{i=1}^{m} (y_i ...
1
vote
1answer
44 views

How does having different scales on features make an elliptical contour plot?

I have been taking Andrew Ng's Machine Learning course, and in the lesson on feature scaling's effect on gradient descent, I just can't understand how because of the different scales on the features ...
2
votes
1answer
79 views

Why am I not getting the correct output from my gradient descent algorithm? [closed]

I have started taking online ML classes, and i was introduced to the topic of Gradient Descent, the Prof, himself hadnt shown us himself how to implement it in a programming language, so for fun, i ...
1
vote
0answers
70 views

Normalized steepest descent with nuclear/frobenius norm

In steepest gradient descent, we try to find a local minima to a loss function $f(\cdot)$ by the rule: $x_{t} = x - \alpha \triangledown_{x}f(x)$. I've found in textbooks that often we want to ...
2
votes
0answers
188 views

How to set the tolerance in Gradient descent?

I understand that one solution of setting the number of iterations, is to set it to a large number and then interrupt it when the gradient vector becomes tiny, so tiny that it is smaller than a ...

1 2
3
4 5
17