# Is this "stochastic gradient method" different from the stochastic gradient descent algorithm?

In a computational statistics book, I found an optimization method to find local minimum of a function.

Let's assume that we have a differentiable function $$f: \mathbb{R}^2 \longrightarrow \mathbb{R}$$. We want to find a local minimum $$f(\theta)$$ of the function. Starting with initialization $$\theta^{*}_i$$, instead of using gradient descent, a slightly different method is used.

Given a decreasing sequence of positive real numbers $$\alpha_i$$ (i = 1,2,3,...)

Set i = 1

1. Sample $$\omega_i$$ from a uniform unit sphere
2. Estimate the directional derivative of $$f$$ at $$\theta = \theta^{*}_i$$ along the direction $$\omega_i$$, denoting the result as $$f_{\omega_i}^{'}( \theta^{*}_i)$$
3. Update $$\theta^{*}_{i+1}$$:= $$\theta^{*}_{i} - \alpha_if_{\omega_i}^{'}( \theta^{*}_i)\omega_i$$
4. Stop and return $$\theta^{*}_{i+1}$$if(convergent). Otherwise increase i by 1 and go back to (1)

This question is mainly because of confusing terminology in different books. I think this algorithm is not the traditional "stochastic gradient descent" algorithm that machine learning scientists often refer to.

Is this true? And if it is, how do statisticians/machine learning scientists often call it?

• Hi! Care to share the book? Anyways, the difference between this algo and stochastic gradient descent is that this chooses a direction completely at random and then goes either in that direction or in the opposite direction according to whichever decreases the cost, whereas stochastic gradient descent goes in a direction which is also sampled at random but crucially is not uniform, and in fact has expectation given by the "true" gradient (i.e. in finite sum applications, the gradient of the entire sum). Dec 4, 2022 at 0:25
• @JohnMadden I will (I forgot the name of that book. The question just popped up today in my mind) Dec 4, 2022 at 0:59

I don't know the name for the algorithm that you mention. It seems like stochastic hill climbing which selects random directions or (random) coordinate descent, which uses steps based on the derivative/gradient, but only along the direction of a single (randomly picked) coordinate.

I imagine that the advantage of changing the angle would be to avoid zigzag patterns that may occur in gradient descent. Stochastic gradient descent refers to the use of randomly optimizing the cost function where the randomness is created by randomly selecting only a part of the data for each step.

Descent in small steps along a complex surface. The cost function is computed for all the data together.  