Is this "stochastic gradient method" different from the stochastic gradient descent algorithm?

Question

In a computational statistics book, I found an optimization method to find local minimum of a function.

Let's assume that we have a differentiable function $f: \mathbb{R}^2 \longrightarrow \mathbb{R}$. We want to find a local minimum $f(\theta)$ of the function. Starting with initialization $\theta^{*}_i$, instead of using gradient descent, a slightly different method is used.

Given a decreasing sequence of positive real numbers $\alpha_i$ (i = 1,2,3,...)

Set i = 1

1. Sample $\omega_i$ from a uniform unit sphere
2. Estimate the directional derivative of $f$ at $\theta = \theta^{*}_i$ along the direction $\omega_i$, denoting the result as $f_{\omega_i}^{'}( \theta^{*}_i)$
3. Update $\theta^{*}_{i+1}$:= $\theta^{*}_{i} - \alpha_if_{\omega_i}^{'}( \theta^{*}_i)\omega_i$
4. Stop and return $\theta^{*}_{i+1}$if(convergent). Otherwise increase i by 1 and go back to (1)

This question is mainly because of confusing terminology in different books. I think this algorithm is not the traditional "stochastic gradient descent" algorithm that machine learning scientists often refer to.

Is this true? And if it is, how do statisticians/machine learning scientists often call it?

Answer 1

I don't know the name for the algorithm that you mention. It seems like stochastic hill climbing which selects random directions or (random) coordinate descent, which uses steps based on the derivative/gradient, but only along the direction of a single (randomly picked) coordinate.

I imagine that the advantage of changing the angle would be to avoid zigzag patterns that may occur in gradient descent.

---

Stochastic gradient descent refers to the use of randomly optimizing the cost function where the randomness is created by randomly selecting only a part of the data for each step.

'regular' gradient descent

Descent in small steps along a complex surface. The cost function is computed for all the data together.

stochastic gradient descent

Descent in small steps along a simple surface, but each time based on a different (random) set of data points.

In the image below you see the paths of descent for six different starting values. The four images relate to four different data points in the (synthetic) data, which creatie different functions to optimise. The steps in the six paths are each time along one of those four surfaces (which of the four is randomly selected each step).