# Does this gradient descent make sense?

I am trying to maximize a function $f(x)$ of a vector of reals $x=<x_1, x_2, ...x_n>$. In my practical application, I have no expression for $f$ whatsoever, all I can do is given a vector $x$, calculate $f(x)$ via a deterministic experiment.

So, instead of having to set a step size $\alpha$, calculate derivatives, etc, just see whether the move $x_1 ->x_1(1+\alpha)$ increases $f$, or $x_1 ->x_1(1-\alpha)$ increases it, transform $x_1$ accordingly, and then iterate over all coordinates in this manner until some termination criterium is met. Here $\alpha$ is a small positive number like $0.05$.

This is more practical for me than a standard gradient descent, because the value of $f$ is completely different order of magnitude of the $x_i$s, as well its derivative, so it will be harder to choose appropriate step size in a classical text book gradient descent manner.

Does this approach make sense? Does it have a specific name?

A note on step size: Generally for gradient descent type algorithms, you want to choose an additive step size which decreases to zero as the optimization proceeds. That is, if moving in the positive direction, set $$x_1\gets x_1 + \mu(1-\epsilon)^n,$$ where $\mu$ is your initial step size, and $\epsilon \ll 1$ is the decay. As $n$ gets large, your algorithm will converge to a specific value of $x_1$. In your post you use a multiplicative step size, which can be dangerous especially if $\alpha$ is not adaptive. For example, suppose that your function is a Gaussian with mean 100, and you initialize $x = 1$ with $\alpha = 0.2$. Then your algorithm will dance around the optimum, like this: Note the much better performance of an additive step with decay (in orange)
• Coordinate descent I think simply refers to breaking the optimization problem into $n$ smaller, single-variable optimization problems. You don't need a grid, meaning the step size $\delta$ could be decreasing as time goes on. I think using a multiplicative step as in your post will give you some unstable behavior, especially if one of the optima has a large coordinate value. – combo Jun 20 '17 at 3:59