Optimization through 'alternating' descent

Question

While analysing a particular iterative method which reaches the fixed points of a multivariate function f(x, y) (x and y are N-dimensional vectors in this case), I was able to reformulate it in the following terms

that is, as some kind of 'alternating' descent, in which x goes in the direction which minimizes y, and vice-versa. Has anyone ever seen something like that, e.g. some algorithm which performs this kind of update?

Actually the 2nd equation has a + instead of a -, so one equation would be performing a descent while the other one performs a ascent... But I am interested in any kind of algorithm which does this kind of alternation.

Answer 1

Take any multivariate function. Doing alternating coordinate descent is a very natural choice and it is being used in many other problems (though not necessarily in the perceptron updates)

Answer 2

This sort of technique us commonly called coordinate descent in machine learning, and can be used to fit a variety of models, IIRC it is guaranteed to converge to the optimal solution provided the cost function is convex. It has been used with SVMs, linear regression, logistic regression, with L2 or L1 penalty functions etc.

A related problem is optimising the hyper-parameters of a model (kernel parameters or regularisation parameters etc.) where the process can be viewed as alternate optimisation of model and hyper-parameters, see Gradient-Based Optimization of Hyperparameters by Yoshua Bengio