# Tag Info

### For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

They say an image is worth more than a thousand words. In the following example (courtesy of MS Paint, a handy tool for amateur and professional statisticians both) you can see a convex function ...
• 11.5k
Accepted

### How could stochastic gradient descent save time compared to standard gradient descent?

Short answer: In many big data setting (say several million data points), calculating cost or gradient takes very long time, because we need to sum over all data points. We do NOT need to have exact ...
• 37.1k
Accepted

### Who invented stochastic gradient descent?

Stochastic Gradient Descent is preceded by Stochastic Approximation as first described by Robbins and Monro in their paper, A Stochastic Approximation Method. Kiefer and Wolfowitz subsequently ...
• 1,842

### For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Gradient descent methods use the slope of the surface. This will not necessarily (or even most likely not) point directly towards the extreme point. An intuitive view is to imagine a path of descent ...
• 81.2k
Accepted

### Why second order SGD convergence methods are unpopular for deep learning?

Should we go toward second order methods for deep learning? TL;DR: No, especially now when the pace of innovation is slowing down, and we're seeing less new architectural innovations, and more ways ...
• 18.1k

### How can it be trapped in a saddle point?

Take a look at the image below from Off Convex. In a convex function (leftmost image), there is only one local minimum, which is also the global minimum. But in a non-convex function (rightmost image),...
• 839
Accepted

The following assumes a loss function $f$ that's expressed as a sum, not an average. Expressing the loss as an average means that the scaling $\frac{1}{n}$ is "baked in" and no further ...
• 92.3k
Accepted

### How does batch size affect convergence of SGD and why?

Sure one update with a big minibatch is "better" (in terms of accuracy) than one update with a small minibatch. This can be seen in the table you copied in your question (call $N$ the sample size): ...
• 8,647

### For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Steepest descent can be inefficient even if the objective function is strongly convex. Ordinary gradient descent I mean "inefficient" in the sense that steepest descent can take steps that oscillate ...
• 92.3k

As other answer suggests, the main reason to use SGD is to reduce the computation cost of gradient while still largely maintaining the gradient direction when averaged over many mini-batches or ...
• 591

### How to choose between SGD with Nesterov momentum and Adam?

In general, there aren't definitive results on one learning algorithm being "better" than another. The common wisdom (which needs to be taken with a pound of salt) has been that Adam requires less ...
• 371

### Who invented stochastic gradient descent?

See Rosenblatt F. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological review. 1958 Nov;65(6):386. I am not sure if SGD was invented ...
• 5,740

### How to set mini-batch size in SGD in keras

Yes you are right. In Keras batch_size refers to the batch size in Mini-batch Gradient Descent. If you want to run a Batch Gradient Descent, you need to set the <...

### For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Local steepest direction is not the same with the global optimum direction. If it were, then your gradient direction wouldn't change; because if you go towards your optimum always, your direction ...
• 57.7k

### RMSProp and Adam vs SGD

After researching a few articles online and Keras documentation it is suggested that the RMSProp optimizer is recommended for recurrent neural networks.https://github.com/keras-team/keras/blob/master/...

### Why second order SGD convergence methods are unpopular for deep learning?

This is actually starting to change as recent work are showing the benefit of second order methods specially for NLP problems. Some examples are: "ADAHESSIAN: An Adaptive Second Order Optimizer ...
• 131
Accepted

### Variance of gradient as e.g. in SGD

I gave this issue some more thoughts and came to the following conclusion: Most of the papers that deal with variance reduction for SGD (methods such as SVRG, SAGA, and SAG) actually mean the 1-norm ...
• 575
Accepted

### Does Keras SGD optimizer implement batch, mini-batch, or stochastic gradient descent?

It works just as you suggest. batch_size parameter does exactly what you would expect: it sets the size of the batch: batch_size: ...
• 11.5k
Accepted

### Can small SGD batch size lead to faster overfitting?

Using "epoch" on the x-axis to compare "speed" of convergence while comparing different batch sizes makes no sense since the number of weight updates per epoch depends on the batch size. When using ...
• 11.5k

### No change in accuracy using Adam Optimizer when SGD works fine

The benefits of Adam can be marginal, at best. The initial results were strong, but there is evidence that Adam converges to dramatically different minima compared to SGD (or SGD + momentum). "The ...
• 92.3k
Accepted

### What should we do when changing SGD optimizer to Adam optimizer?

In my experience, changing optimizers is not a simple matter of swapping one for the other. Instead, changing optimizers also interacts with several other configuration choices in the neural network. ...
• 92.3k
Accepted

• 26.3k
Accepted

### For regression with time varying parameters, SGD or Kalman filter?

Both of these things can be used in an online manner, but they do this in different ways. So they are not competitors. The Kalman filter has two purposes. First, for a batch of data, it will yield ...
• 20.9k