# What happened to the "other" stochastic gradient descent?

Stochastic gradient descent is a useful approach to improving iteration time by giving up some rate of convergence. For a parameter $$w$$, learning rate $$\eta$$, and smooth objective function $$Q$$ the update rule is:

The term containing "stochastic" reminded me of stochastic processes and their role in SDEs and numerical algorithms like stochastic Runge-Kutta.

The idea of having an iterative algorithm which adds instances of a stochastic process seems like a possibility that someone would have explored by now for gradient descent. One could begin with modifying vanilla gradient descent by adding elements from a stochastic process to it:

$$W_{n+1} = W_n - \eta \nabla Q(w) + X_n$$

where $$X_n$$ is the $$n$$th element of a stochastic process.

Not all choices of stochastic process would result in the update rule having some form of convergence in the parameter random variable. Indeed, on the face of it including noise like this could seem like a horrible idea since you'll get inferior rate of convergence and makes the final parameter estimate noisier.

The main benefit I see is to allow jumps in the parameter coordinate to help avoid to getting stuck in local minima. Naturally, there are other approaches like adaptive learning rates to tackle this.

Maybe such alternatives are generally better, but for me this other "stochastic gradient descent" is an under-examined possibility. I imagine the approach isn't popular for the inefficiencies it introduces, but I can also imagine that for sufficiently gnarly parameters spaces (e.g. lots of local minima) that its ability to make jumps could be helpful.

Is this idea a dead end that was already explored in the literature? Or is it a fruitful niche that exists in the literature? Or is it still off people's radar?

• SGD also has the benefit of allowing jumps (it is an effective way of adding noise), while requiring much less computations (which is one of the the main motivations of using it). Computing the full gradient and then adding noise seems a bit strange from that perspective Apr 23, 2023 at 18:28
• @J.Delaney - it seems similar to "exploration vs. exploitation" thinking to me; I don't know that you would want to do it on every step, but... Apr 23, 2023 at 18:38
• @jbowman The exploration vs exploitation tradeoff is what I was thinking about, but the term is often used in reinforcement learning. I thought the term might confuse the scope of the question. Apr 23, 2023 at 18:40
• optimization in the context of ML / neural networks (where SGD is most common) is a bit different from classical optimization in that trying "too hard" to find a lower local minimum is generally considered a bad idea, as it often leads to overfitting. Apr 23, 2023 at 18:56
• @J.Delaney That's an excellent point. Thinking about various test functions led me to think about the benefit of hopping around more. But it is also true that the loss surface is subject to sampling error and measurement error. Some optima may be stable across samples, and others not. Definitely worth some more thought on my part. Of course in practice we have to test our models! Apr 23, 2023 at 19:02

• I'm merely implying that the main motivation for SGD is the speed. I think you could argue that momentum-based SGD is doing exactly what you are saying, i.e. altering the form of the noise to avoid local minimum. If you mean adding new noise explicitly (i.e. +rand(...)) rather than implicitly (sub-sampling the data), I'm guessing this isn't very much explored due to being a bit counter intuitive. If you could show that adding a noise term is better than a momentum based approach, I think you would surprise a lot of researchers. Apr 23, 2023 at 20:40