Questions tagged [stochastic-gradient-descent]

Filter by
Sorted by
Tagged with
6
votes
1answer
1k views

Mean or sum of gradients for weight updates in SGD

I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
25
votes
2answers
13k views

How could stochastic gradient descent save time compared to standard gradient descent?

Standard Gradient Descent would compute gradient for the entire training dataset. ...
134
votes
4answers
130k views

Batch gradient descent versus stochastic gradient descent

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
26
votes
6answers
7k views

For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
19
votes
2answers
4k views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
4
votes
2answers
2k views

Stochastic gradient descent Vs Mini-batch size 1

Is stochastic gradient descent basically the name given to mini-batch training where batch size = 1 and selecting random training rows? i.e. it is the same as 'normal' gradient descent, it's just the ...
16
votes
4answers
12k views

How can it be trapped in a saddle point?

I am currently a bit puzzled by how mini-batch gradient descent can be trapped in a saddle point. The solution might be too trivial that I don't get it. You get an new sample every epoch, and it ...
2
votes
1answer
6k views

No change in accuracy using Adam Optimizer when SGD works fine

I have been training a Spatial Transformer network with DNN on GTRSB dataset. I initially used SGF with momentum and was able to achieve good accuracy. For further improvements and testing, I ...
8
votes
1answer
2k views

Does Keras SGD optimizer implement batch, mini-batch, or stochastic gradient descent?

I am a newbie in Deep Learning libraries and thus decided to go with Keras. While implementing a NN model, I saw the batch_size parameter in ...
26
votes
4answers
34k views

How does batch size affect convergence of SGD and why?

I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
13
votes
2answers
24k views

How to set mini-batch size in SGD in keras

I am new to Keras and need your help. I am training a neural net in Keras and my loss function is Squared Difference b/w net's output and target value. I want to optimize this using Gradient Descent....
10
votes
3answers
5k views

Gradient descent on non-convex functions

What situations do we know of where gradient descent can be shown to converge (either to a critical point or to a local/global minima) for non-convex functions? For SGD on non-convex functions, one ...
2
votes
1answer
363 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
1answer
137 views

Trees of ensembles.

I have a large dataset (100k+), and it's growing everyday. I want to train it to predict a value (a regression problem). I've been finding that ensemble trees work the best for now, but in the ...
3
votes
1answer
80 views

The correct implementation of momentum method and NAG

Recently started a Coursera course on Deep Learning. In the optimization video, momentum and NAG were not very clearly explained so, I searched and came across the paper On the importance of ...
2
votes
1answer
150 views

Matrix factorization for expanding matrix

In the paper Matrix Factorization Techniques for Recommender Systems Koren, Bell and Volinsky describe how the matrix $R_{n \times k}$ (users $\times$ movie ratings) can be decomposed to $P_{n \times ...
1
vote
1answer
273 views

What is neural network good accuracy

I am very new at machine learning, and I'm building an artificial neural network that aims to classify inputs into 2 labels. I am training the network with randomly initialized weights and through the ...
1
vote
0answers
74 views

Momentum updates average of $g$, Adagrad also of $g^2$ - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
1
vote
0answers
135 views

What cause $X\beta$ shift from Stochastic Gradient Descent Comparing to Logistic Regression?

I am experimenting with stochastic gradient descent and observing very strange output. In a toy problem, the $X\beta$ for stochastic gradient descent is always larger than $0$, which will be ...