Questions tagged [stochastic-gradient-descent]
The stochastic-gradient-descent tag has no usage guidance.
23 questions
11
votes
1
answer
6k
views
Mean or sum of gradients for weight updates in SGD
I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
- 343
33
votes
3
answers
22k
views
How could stochastic gradient descent save time compared to standard gradient descent?
Standard Gradient Descent would compute gradient for the entire training dataset.
...
- 1,175
154
votes
7
answers
175k
views
Batch gradient descent versus stochastic gradient descent
Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
- 1,581
30
votes
6
answers
11k
views
For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?
Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process.
My question is, given the point on the convex, does the ...
- 2,485
33
votes
2
answers
14k
views
Why second order SGD convergence methods are unpopular for deep learning?
It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/
They trace ...
- 431
7
votes
2
answers
4k
views
Stochastic gradient descent Vs Mini-batch size 1
Is stochastic gradient descent basically the name given to mini-batch training where batch size = 1 and selecting random training rows? i.e. it is the same as 'normal' gradient descent, it's just the ...
- 464
22
votes
4
answers
22k
views
How can it be trapped in a saddle point?
I am currently a bit puzzled by how mini-batch gradient descent can be trapped in a saddle point.
The solution might be too trivial that I don't get it.
You get an new sample every epoch, and it ...
- 321
20
votes
3
answers
9k
views
When will gradient descent converge to a critical point or to a local/global minima) for non-convex functions?
What situations do we know of where gradient descent can be shown to converge (either to a critical point or to a local/global minima) for non-convex functions?
For SGD on non-convex functions, one ...
- 361
5
votes
1
answer
10k
views
No change in accuracy using Adam Optimizer when SGD works fine
I have been training a Spatial Transformer network with DNN on GTRSB dataset. I initially used SGF with momentum and was able to achieve good accuracy.
For further improvements and testing, I ...
- 151
2
votes
1
answer
383
views
Matrix factorization for expanding matrix
In the paper Matrix Factorization Techniques for Recommender Systems Koren, Bell and Volinsky describe how the matrix $R_{n \times k}$ (users $\times$ movie ratings) can be decomposed to $P_{n \times ...
- 141k
10
votes
1
answer
4k
views
Does Keras SGD optimizer implement batch, mini-batch, or stochastic gradient descent?
I am a newbie in Deep Learning libraries and thus decided to go with Keras. While implementing a NN model, I saw the batch_size parameter in ...
- 225
46
votes
2
answers
24k
views
Who invented stochastic gradient descent?
I'm trying to understand the history of Gradient descent and Stochastic gradient descent. Gradient descent was invented in Cauchy in 1847.Méthode générale pour la résolution des systèmes d'équations ...
- 4,732
36
votes
4
answers
50k
views
How does batch size affect convergence of SGD and why?
I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
- 16.8k
32
votes
4
answers
66k
views
How can Stochastic Gradient Descent(SGD) avoid the problem of local minima?
I know that Stochastic Gradient Descent(SGD) has random behavior, but I don't know why.
Is there any explanation about this?
- 513
16
votes
2
answers
37k
views
How to set mini-batch size in SGD in keras
I am new to Keras and need your help.
I am training a neural net in Keras and my loss function is Squared Difference b/w net's output and target value.
I want to optimize this using Gradient Descent....
- 171
3
votes
1
answer
979
views
Relationship between variance of gradient and SGD convergence
I've found things such as the Robbins-Monroe conditions for the learning rate, as well as a proof from Robbins, Siegmund, 1971 which gives convergence to a local minima provided that the expectation ...
- 380
3
votes
1
answer
889
views
Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively repel them?
While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
- 431
3
votes
1
answer
223
views
The correct implementation of momentum method and NAG
Recently started a Coursera course on Deep Learning. In the optimization video, momentum and NAG were not very clearly explained so, I searched and came across the paper On the importance of ...
- 33
2
votes
1
answer
3k
views
What is neural network good accuracy
I am very new at machine learning, and I'm building an artificial neural network that aims to classify inputs into 2 labels. I am training the network with randomly initialized weights and through the ...
- 583
1
vote
0
answers
144
views
Momentum updates average of $g$, Adagrad also of $g^2$ - any other interesting updated averages for SGD convergence?
Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics.
Then e.g. Adagrad, ADAM family ...
- 431
1
vote
1
answer
160
views
Trees of ensembles.
I have a large dataset (100k+), and it's growing everyday. I want to train it to predict a value (a regression problem).
I've been finding that ensemble trees work the best for now, but in the ...
- 826
1
vote
0
answers
173
views
What cause $X\beta$ shift from Stochastic Gradient Descent Comparing to Logistic Regression?
I am experimenting with stochastic gradient descent and observing very strange output.
In a toy problem, the $X\beta$ for stochastic gradient descent is always larger than $0$, which will be ...
- 37.3k
1
vote
1
answer
959
views
Should I use the whole dataset in the forward pass when doing minibatch gradient descent?
I've implemented the following algorithm. For each minibatch:
Compute the gradient using the mini-batch sample
Update the parameters
Update the hidden layers. If $\Gamma_L$ are the new parameters ...
- 13.7k